From 8ba320c04d1a6bbe4a655fa3c38d8df052d4f89e Mon Sep 17 00:00:00 2001 From: don sizemore Date: Mon, 25 Jul 2022 13:05:01 -0400 Subject: [PATCH 001/578] #8856 install on RHEL/Rocky 9 --- .../source/installation/prerequisites.rst | 36 +++++++++++-------- .../source/installation/shibboleth.rst | 6 ++-- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index 0ad3bf600c9..afc0f2bbce4 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -14,7 +14,7 @@ After following all the steps below, you can proceed to the :doc:`installation-m Linux ----- -We assume you plan to run your Dataverse installation on Linux and we recommend RHEL or a derivative such as RockyLinux or AlmaLinux, which is the distribution family tested by the Dataverse Project team. Please be aware that while EL8 (RHEL/derivatives) is the recommended platform, the steps below were orginally written for EL6 and may need to be updated (please feel free to make a pull request!). A number of community members have installed the Dataverse Software in Debian/Ubuntu environments. +We assume you plan to run your Dataverse installation on Linux and we recommend RHEL or a derivative such as RockyLinux or AlmaLinux, which is the distribution family tested by the Dataverse Project team. These instructions are written for RHEL9 and derivatives with notes for RHEL 8 and earlier, but a number of community members have successfully installed Dataverse in Debian and Ubuntu environments. Java ---- @@ -28,9 +28,9 @@ The Dataverse Software should run fine with only the Java Runtime Environment (J The Oracle JDK can be downloaded from http://www.oracle.com/technetwork/java/javase/downloads/index.html -On a RHEL/derivative, install OpenJDK (devel version) using yum:: +On a RHEL/derivative OS, install OpenJDK (devel version) using dnf:: - # sudo yum install java-11-openjdk + # sudo dnf install java-11-openjdk If you have multiple versions of Java installed, Java 11 should be the default when ``java`` is invoked from the command line. You can test this by running ``java -version``. @@ -98,16 +98,16 @@ PostgreSQL Installing PostgreSQL ===================== -The application has been tested with PostgreSQL versions up to 13 and version 10+ is required. We recommend installing the latest version that is available for your OS distribution. *For example*, to install PostgreSQL 13 under RHEL7/derivative:: +The application has been tested with PostgreSQL versions up to 13 and version 10+ is required. We recommend installing the latest version that is available for your OS distribution. *For example*, to install PostgreSQL 13 under RHEL9/derivative:: - # yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm - # yum makecache fast - # yum install -y postgresql13-server - # /usr/pgsql-13/bin/postgresql-13-setup initdb - # /usr/bin/systemctl start postgresql-13 - # /usr/bin/systemctl enable postgresql-13 + # sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm + # sudo dnf makecache + # sudo dnf install -y postgresql13-server + # sudo /usr/pgsql-13/bin/postgresql-13-setup initdb + # sudo /usr/bin/systemctl start postgresql-13 + # sudo /usr/bin/systemctl enable postgresql-13 -For RHEL8/derivative the process would be identical, except for the first two commands: you would need to install the "EL-8" yum repository configuration and run ``yum makecache`` instead. +For RHEL8/derivative the process would be identical but requires a disabling the OS' built-in postgresql module with ``dnf -qy module disable postgresql``. On RHEL7 the second command should be ``yum makecache fast``. Configuring Database Access for the Dataverse Installation (and the Dataverse Software Installer) ================================================================================================= @@ -141,7 +141,7 @@ Configuring Database Access for the Dataverse Installation (and the Dataverse So The file ``postgresql.conf`` will be located in the same directory as the ``pg_hba.conf`` above. -- **Important: PostgreSQL must be restarted** for the configuration changes to take effect! On RHEL7/derivative and similar (provided you installed Postgres as instructed above):: +- **Important: PostgreSQL must be restarted** for the configuration changes to take effect! On RHEL9/derivative and similar (provided you installed Postgres as instructed above):: # systemctl restart postgresql-13 @@ -307,14 +307,22 @@ For RHEL/derivative, the EPEL distribution is strongly recommended: If :fixedwidthplain:`yum` isn't configured to use EPEL repositories ( https://fedoraproject.org/wiki/EPEL ): +RHEL9/derivative users can install the epel-release RPM:: + + dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm + RHEL8/derivative users can install the epel-release RPM:: - yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm + dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm RHEL7/derivative users can install the epel-release RPM:: yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm +Rocky 9 users will need te enable the CodeReady-Builder repository:: + + dnf config-manager --enable crb + RHEL 8 users will need to enable the CodeReady-Builder repository:: subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms @@ -449,7 +457,7 @@ As root, create a "counter" user and change ownership of Counter Processor direc Installing Counter Processor Python Requirements ================================================ -Counter Processor version 0.1.04 requires Python 3.7 or higher. This version of Python is available in many operating systems, and is purportedly available for RHEL7 or CentOS 7 via Red Hat Software Collections. Alternately, one may compile it from source. +Counter Processor version 0.1.04 requires Python 3.7 or higher. This version of Python or higher is available in many operating systems, and is purportedly available for RHEL7 or CentOS 7 via Red Hat Software Collections. Alternately, one may compile it from source. The following commands are intended to be run as root but we are aware that Pythonistas might prefer fancy virtualenv or similar setups. Pull requests are welcome to improve these steps! diff --git a/doc/sphinx-guides/source/installation/shibboleth.rst b/doc/sphinx-guides/source/installation/shibboleth.rst index cd0fbda77a6..b7379bece49 100644 --- a/doc/sphinx-guides/source/installation/shibboleth.rst +++ b/doc/sphinx-guides/source/installation/shibboleth.rst @@ -32,7 +32,7 @@ We will be "fronting" the app server with Apache so that we can make use of the We include the ``mod_ssl`` package to enforce HTTPS per below. -``yum install httpd mod_ssl`` +``dnf install httpd mod_ssl`` Install Shibboleth ~~~~~~~~~~~~~~~~~~ @@ -51,7 +51,7 @@ Install Shibboleth Via Yum Please note that during the installation it's ok to import GPG keys from the Shibboleth project. We trust them. -``yum install shibboleth`` +``dnf install shibboleth`` Configure Payara ---------------- @@ -122,7 +122,7 @@ Near the bottom of ``/etc/httpd/conf.d/ssl.conf`` but before the closing ``` to compare it against the file you edited. -Note that ``/etc/httpd/conf.d/shib.conf`` and ``/etc/httpd/conf.d/shibboleth-ds.conf`` are expected to be present from installing Shibboleth via yum. +Note that ``/etc/httpd/conf.d/shib.conf`` and ``/etc/httpd/conf.d/shibboleth-ds.conf`` are expected to be present from installing Shibboleth via dnf. You may wish to also add a timeout directive to the ProxyPass line within ssl.conf. This is especially useful for larger file uploads as apache may prematurely kill the connection before the upload is processed. From b22ecb728edbece86c83803b03fd0ab9330364cd Mon Sep 17 00:00:00 2001 From: don sizemore Date: Mon, 25 Jul 2022 13:08:16 -0400 Subject: [PATCH 002/578] #8856 convert Solr, R and JQ yum commands to dnf --- .../source/installation/prerequisites.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index afc0f2bbce4..48dffc2aacd 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -199,7 +199,7 @@ On operating systems which use systemd such as RHEL/derivative, you may then add Solr launches asynchronously and attempts to use the ``lsof`` binary to watch for its own availability. Installation of this package isn't required but will prevent a warning in the log at startup:: - # yum install lsof + # dnf install lsof Finally, you need to tell Solr to create the core "collection1" on startup:: @@ -255,8 +255,8 @@ Installing jq ``jq`` is a command line tool for parsing JSON output that is used by the Dataverse Software installation script. It is available in the EPEL repository:: - # yum install epel-release - # yum install jq + # dnf install epel-release + # dnf install jq or you may install it manually:: @@ -277,7 +277,7 @@ Installing and configuring ImageMagick On a Red Hat or derivative Linux distribution, you can install ImageMagick with something like:: - # yum install ImageMagick + # dnf install ImageMagick (most RedHat systems will have it pre-installed). When installed using standard ``yum`` mechanism, above, the executable for the ImageMagick convert utility will be located at ``/usr/bin/convert``. No further configuration steps will then be required. @@ -305,7 +305,7 @@ Installing R For RHEL/derivative, the EPEL distribution is strongly recommended: -If :fixedwidthplain:`yum` isn't configured to use EPEL repositories ( https://fedoraproject.org/wiki/EPEL ): +If :fixedwidthplain:`dnf` isn't configured to use EPEL repositories ( https://fedoraproject.org/wiki/EPEL ): RHEL9/derivative users can install the epel-release RPM:: @@ -336,9 +336,9 @@ RHEL 7 users will want to log in to their organization's respective RHN interfac • click on "Subscribed Channels: Alter Channel Subscriptions" • enable EPEL, Server Extras, Server Optional -Finally, install R with :fixedwidthplain:`yum`:: +Finally, install R with :fixedwidthplain:`dnf`:: - yum install R-core R-core-devel + dnf install R-core R-core-devel Installing the required R libraries =================================== @@ -463,7 +463,7 @@ The following commands are intended to be run as root but we are aware that Pyth Install Python 3.9:: - yum install python39 + dnf install python39 Install Counter Processor Python requirements:: From a9f72f2b72f96a5a4414af4940151ec3c109251b Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 11 Aug 2022 14:49:39 -0400 Subject: [PATCH 003/578] HDC 3b 1.5 initial update --- .../harvard/iq/dataverse/MailServiceBean.java | 4 +- .../harvard/iq/dataverse/api/LDNInbox.java | 132 +++++----- .../LDNAnnounceDatasetVersionStep.java | 246 +++++++++--------- 3 files changed, 177 insertions(+), 205 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java index fa5216140c2..084a7058e70 100644 --- a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java @@ -621,9 +621,9 @@ public String getMessageTextBasedOnNotification(UserNotification userNotificatio Object[] paramArrayDatasetMentioned = { userNotification.getUser().getName(), BrandingUtil.getInstallationBrandName(), - citingResource.getString("@type"), + citingResource.getString("@type", "External Resource"), citingResource.getString("@id"), - citingResource.getString("name"), + citingResource.getString("name", citingResource.getString("@id")), citingResource.getString("relationship"), systemConfig.getDataverseSiteUrl(), dataset.getGlobalId().toString(), diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index 3912b9102e2..fabc067abf7 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -73,8 +73,6 @@ public Response acceptMessage(String body) { String whitelist = settingsService.get(SettingsServiceBean.Key.LDNMessageHosts.toString(), ""); // Only do something if we listen to this host if (whitelist.equals("*") || whitelist.contains(origin.toString())) { - String citingPID = null; - String citingType = null; boolean sent = false; JsonObject jsonld = null; @@ -106,85 +104,73 @@ public Response acceptMessage(String body) { String objectKey = new JsonLDTerm(activityStreams, "object").getUrl(); if (jsonld.containsKey(objectKey)) { JsonObject msgObject = jsonld.getJsonObject(objectKey); - - citingPID = msgObject.getJsonObject(new JsonLDTerm(ietf, "cite-as").getUrl()).getString("@id"); - logger.fine("Citing PID: " + citingPID); - if (msgObject.containsKey("@type")) { - citingType = msgObject.getString("@type"); - if (citingType.startsWith(JsonLDNamespace.schema.getUrl())) { - citingType = citingType.replace(JsonLDNamespace.schema.getUrl(), ""); - } - if (msgObject.containsKey(JsonLDTerm.schemaOrg("name").getUrl())) { - name = msgObject.getString(JsonLDTerm.schemaOrg("name").getUrl()); - } - logger.fine("Citing Type: " + citingType); - String contextKey = new JsonLDTerm(activityStreams, "context").getUrl(); - - if (jsonld.containsKey(contextKey)) { - JsonObject context = jsonld.getJsonObject(contextKey); - for (Map.Entry entry : context.entrySet()) { - - relationship = entry.getKey().replace("_:", ""); - // Assuming only one for now - should check for array and loop - JsonObject citedResource = (JsonObject) entry.getValue(); - String pid = citedResource.getJsonObject(new JsonLDTerm(ietf, "cite-as").getUrl()) - .getString("@id"); - if (citedResource.getString("@type").equals(JsonLDTerm.schemaOrg("Dataset").getUrl())) { - logger.fine("Raw PID: " + pid); - if (pid.startsWith(GlobalId.DOI_RESOLVER_URL)) { - pid = pid.replace(GlobalId.DOI_RESOLVER_URL, GlobalId.DOI_PROTOCOL + ":"); - } else if (pid.startsWith(GlobalId.HDL_RESOLVER_URL)) { - pid = pid.replace(GlobalId.HDL_RESOLVER_URL, GlobalId.HDL_PROTOCOL + ":"); - } - logger.fine("Protocol PID: " + pid); - Optional id = GlobalId.parse(pid); - Dataset dataset = datasetSvc.findByGlobalId(pid); - if (dataset != null) { - JsonObject citingResource = Json.createObjectBuilder().add("@id", citingPID) - .add("@type", citingType).add("relationship", relationship) - .add("name", name).build(); - StringWriter sw = new StringWriter(128); - try (JsonWriter jw = Json.createWriter(sw)) { - jw.write(citingResource); - } - String jsonstring = sw.toString(); - Set ras = roleService.rolesAssignments(dataset); - - roleService.rolesAssignments(dataset).stream() - .filter(ra -> ra.getRole().permissions() - .contains(Permission.PublishDataset)) - .flatMap( - ra -> roleAssigneeService - .getExplicitUsers(roleAssigneeService - .getRoleAssignee(ra.getAssigneeIdentifier())) - .stream()) - .distinct() // prevent double-send - .forEach(au -> { - - if (au.isSuperuser()) { - userNotificationService.sendNotification(au, - new Timestamp(new Date().getTime()), - UserNotification.Type.DATASETMENTIONED, dataset.getId(), - null, null, true, jsonstring); - - } - }); - sent = true; + if (new JsonLDTerm(activityStreams, "Relationship").getUrl().equals(msgObject.getJsonString("@type"))) { + // We have a relationship message - need to get the subject and object and + // relationship type + String subjectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "subject").getUrl()) + .getString("@id"); + String objectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "object").getUrl()) + .getString("@id"); + String relationshipId = msgObject + .getJsonObject(new JsonLDTerm(activityStreams, "relationship").getUrl()).getString("@id"); + if (subjectId != null && objectId != null && relationshipId != null) { + // Strip the URL part from a relationship ID/URL assuming a usable label exists + // after a # char. Default is to use the whole URI. + relationship = relationshipId.substring(relationship.indexOf("#") + 1); + // Parse the URI as a PID and see if this Dataverse instance has this dataset + String pid = GlobalId.getInternalFormOfPID(objectId); + Optional id = GlobalId.parse(pid); + if (id.isPresent()) { + Dataset dataset = datasetSvc.findByGlobalId(pid); + if (dataset != null) { + JsonObject citingResource = Json.createObjectBuilder().add("@id", subjectId) + .add("relationship", relationship).build(); + StringWriter sw = new StringWriter(128); + try (JsonWriter jw = Json.createWriter(sw)) { + jw.write(citingResource); } + String jsonstring = sw.toString(); + Set ras = roleService.rolesAssignments(dataset); + + roleService.rolesAssignments(dataset).stream() + .filter(ra -> ra.getRole().permissions().contains(Permission.PublishDataset)) + .flatMap(ra -> roleAssigneeService + .getExplicitUsers( + roleAssigneeService.getRoleAssignee(ra.getAssigneeIdentifier())) + .stream()) + .distinct() // prevent double-send + .forEach(au -> { + + if (au.isSuperuser()) { + userNotificationService.sendNotification(au, + new Timestamp(new Date().getTime()), + UserNotification.Type.DATASETMENTIONED, dataset.getId(), null, + null, true, jsonstring); + + } + }); + sent = true; } + } else { + // We don't have a dataset corresponding to the object of the relationship - do + // nothing } + + } else { + // Can't get subject, relationship, object from message + logger.info("Can't find the subject, relationship or object in the message - ignoring"); + } + } else { } + // This isn't a Relationship announcement message - ignore + logger.info("This is not a relationship announcement - ignoring message of type " + + msgObject.getJsonString("@type")); + } if (!sent) { - if (citingPID == null || citingType == null) { - throw new BadRequestException( - "Could not parse message to find acceptable citation link to a dataset."); - } else { - throw new ServiceUnavailableException( - "Unable to process message. Please contact the administrators."); - } + throw new ServiceUnavailableException("Unable to process message. Please contact the administrators."); } } else { logger.info("Ignoring message from IP address: " + origin.toString()); diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java index 3478d9398f0..1b373ffefa8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -18,8 +18,10 @@ import java.net.URI; import java.net.URISyntaxException; import java.nio.charset.StandardCharsets; -import java.util.Collection; +import java.util.Arrays; import java.util.HashMap; +import java.util.HashSet; +import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Map.Entry; @@ -32,6 +34,7 @@ import javax.json.JsonArrayBuilder; import javax.json.JsonObject; import javax.json.JsonObjectBuilder; +import javax.json.JsonString; import javax.json.JsonValue; import org.apache.http.client.methods.CloseableHttpResponse; @@ -56,9 +59,7 @@ public class LDNAnnounceDatasetVersionStep implements WorkflowStep { private static final String LDN_TARGET = ":LDNTarget"; private static final String RELATED_PUBLICATION = "publication"; - JsonLDTerm publicationIDType = null; - JsonLDTerm publicationIDNumber = null; - JsonLDTerm publicationURL = null; + public LDNAnnounceDatasetVersionStep(Map paramSet) { new HashMap<>(paramSet); @@ -77,32 +78,49 @@ public WorkflowStepResult run(WorkflowContext context) { HttpPost announcement; try { - announcement = buildAnnouncement(false, context, target); - } catch (URISyntaxException e) { - return new Failure("LDNAnnounceDatasetVersion workflow step failed: unable to parse inbox in :LDNTarget setting."); - } - if(announcement==null) { - logger.info(context.getDataset().getGlobalId().asString() + "does not have metadata required to send LDN message. Nothing sent."); - return OK; - } - // execute - try (CloseableHttpResponse response = client.execute(announcement)) { - int code = response.getStatusLine().getStatusCode(); - if (code >= 200 && code < 300) { - // HTTP OK range - return OK; - } else { - String responseBody = new String(response.getEntity().getContent().readAllBytes(), - StandardCharsets.UTF_8); - ; - return new Failure("Error communicating with " + inboxUrl + ". Server response: " + responseBody - + " (" + response + ")."); + // First check that we have what is required + Dataset d = context.getDataset(); + DatasetVersion dv = d.getReleasedVersion(); + List dvf = dv.getDatasetFields(); + Map fields = new HashMap(); + List reqFields = Arrays.asList(((String) context.getSettings().getOrDefault(REQUIRED_FIELDS, "")).split(",\\s*")); + for (DatasetField df : dvf) { + if(!df.isEmpty() && reqFields.contains(df.getDatasetFieldType().getName())) { + fields.put(df.getDatasetFieldType().getName(), df); + } + } + //Loop through and send a message for each supported relationship + boolean success=false; + for (JsonObject rel : getObjects(context, fields).getValuesAs(JsonObject.class)) { + announcement = buildAnnouncement(d, rel, target); + // execute + try (CloseableHttpResponse response = client.execute(announcement)) { + int code = response.getStatusLine().getStatusCode(); + if (code >= 200 && code < 300) { + // HTTP OK range + success=true; + logger.fine("Successfully sent message for " + rel.toString()); + } else { + String responseBody = new String(response.getEntity().getContent().readAllBytes(), + StandardCharsets.UTF_8); + ; + return new Failure((success ? "Partial failure":"") + "Error communicating with " + inboxUrl + " for relationship " + rel.toString() +". Server response: " + responseBody + + " (" + response + ")."); + } + + } catch (Exception ex) { + logger.log(Level.SEVERE, "Error communicating with remote server: " + ex.getMessage(), ex); + return new Failure((success ? "Partial failure":"") + "Error executing request: " + ex.getLocalizedMessage(), + "Cannot communicate with remote server."); + } + } + //Any failure and we would have returned already. + return OK; - } catch (Exception ex) { - logger.log(Level.SEVERE, "Error communicating with remote server: " + ex.getMessage(), ex); - return new Failure("Error executing request: " + ex.getLocalizedMessage(), - "Cannot communicate with remote server."); + + } catch (URISyntaxException e) { + return new Failure("LDNAnnounceDatasetVersion workflow step failed: unable to parse inbox in :LDNTarget setting."); } } return new Failure("LDNAnnounceDatasetVersion workflow step failed: :LDNTarget setting missing or invalid."); @@ -118,145 +136,113 @@ public void rollback(WorkflowContext context, Failure reason) { throw new UnsupportedOperationException("Not supported yet."); // This class does not need to resume. } - HttpPost buildAnnouncement(boolean qb, WorkflowContext ctxt, JsonObject target) throws URISyntaxException { - - // First check that we have what is required - DatasetVersion dv = ctxt.getDataset().getReleasedVersion(); - List dvf = dv.getDatasetFields(); - Map fields = new HashMap(); - String[] requiredFields = ((String) ctxt.getSettings().getOrDefault(REQUIRED_FIELDS, "")).split(",\\s*"); - for (String field : requiredFields) { - fields.put(field, null); - } - Set reqFields = fields.keySet(); - for (DatasetField df : dvf) { - if(!df.isEmpty() && reqFields.contains(df.getDatasetFieldType().getName())) { - fields.put(df.getDatasetFieldType().getName(), df); - } - } - if (fields.containsValue(null)) { - logger.fine("DatasetVersion doesn't contain metadata required to trigger announcement"); - return null; - } - // We do, so construct the json-ld body and method - + /**Scan through all fields and return an array of relationship JsonObjects with subjectId, relationship, objectId, and @context + * + * @param ctxt + * @param fields + * @return JsonArray of JsonObjects with subjectId, relationship, objectId, and @context + */ + JsonArray getObjects(WorkflowContext ctxt, Map fields) { + JsonArrayBuilder jab = Json.createArrayBuilder(); Map localContext = new HashMap(); - JsonObjectBuilder coarContext = Json.createObjectBuilder(); Map emptyCvocMap = new HashMap(); - boolean includeLocalContext = false; + + Dataset d = ctxt.getDataset(); for (Entry entry : fields.entrySet()) { DatasetField field = entry.getValue(); DatasetFieldType dft = field.getDatasetFieldType(); - String dfTypeName = entry.getKey(); JsonValue jv = OREMap.getJsonLDForField(field, false, emptyCvocMap, localContext); - switch (dfTypeName) { - case RELATED_PUBLICATION: - JsonArrayBuilder relArrayBuilder = Json.createArrayBuilder(); - publicationIDType = null; - publicationIDNumber = null; - publicationURL = null; - Collection childTypes = dft.getChildDatasetFieldTypes(); - for (DatasetFieldType cdft : childTypes) { - switch (cdft.getName()) { - case "publicationURL": - publicationURL = cdft.getJsonLDTerm(); - break; - case "publicationIDType": - publicationIDType = cdft.getJsonLDTerm(); - break; - case "publicationIDNumber": - publicationIDNumber = cdft.getJsonLDTerm(); - break; + //jv is a JsonArray for multi-val fields, so loop + if (jv != null) { + if (jv instanceof JsonArray) { + JsonArray rels = (JsonArray) jv; + Iterator iter = rels.iterator(); + while(iter.hasNext()) { + JsonValue jval =iter.next(); + jab.add(getRelationshipObject(dft, jval, d, localContext)); } - + } else { + jab.add(getRelationshipObject(dft, jv, d, localContext)); } + } + + } + return jab.build(); + } + + private JsonObject getRelationshipObject(DatasetFieldType dft, JsonValue jval, Dataset d, Map localContext) { + String id = getBestId(dft, jval); + return Json.createObjectBuilder().add("object", id).add("relationship", dft.getJsonLDTerm().getUrl()) + .add("subject", d.getGlobalId().toURL().toString()).build(); + } + - if (jv != null) { - if (jv instanceof JsonArray) { - JsonArray rels = (JsonArray) jv; - for (JsonObject jo : rels.getValuesAs(JsonObject.class)) { - String id = getBestPubId(jo); - relArrayBuilder.add(Json.createObjectBuilder().add("id", id).add("ietf:cite-as", id) - .add("type", "sorg:ScholaryArticle").build()); - } - } + HttpPost buildAnnouncement(Dataset d, JsonObject rel, JsonObject target) throws URISyntaxException { - else { // JsonObject - String id = getBestPubId((JsonObject) jv); - relArrayBuilder.add(Json.createObjectBuilder().add("id", id).add("ietf:cite-as", id) - .add("type", "sorg:ScholaryArticle").build()); - } - } - coarContext.add("IsSupplementTo", relArrayBuilder); - break; - default: - if (jv != null) { - includeLocalContext = true; - coarContext.add(dft.getJsonLDTerm().getLabel(), jv); - } - } - } - dvf.get(0).getDatasetFieldType().getName(); JsonObjectBuilder job = Json.createObjectBuilder(); JsonArrayBuilder context = Json.createArrayBuilder().add("https://purl.org/coar/notify") .add("https://www.w3.org/ns/activitystreams"); - if (includeLocalContext && !localContext.isEmpty()) { - JsonObjectBuilder contextBuilder = Json.createObjectBuilder(); - for (Entry e : localContext.entrySet()) { - contextBuilder.add(e.getKey(), e.getValue()); - } - context.add(contextBuilder); - } job.add("@context", context); job.add("id", "urn:uuid:" + UUID.randomUUID().toString()); job.add("actor", Json.createObjectBuilder().add("id", SystemConfig.getDataverseSiteUrlStatic()) .add("name", BrandingUtil.getInstallationBrandName()).add("type", "Service")); - job.add("context", coarContext); - Dataset d = ctxt.getDataset(); - job.add("object", - Json.createObjectBuilder().add("id", d.getLocalURL()) - .add("ietf:cite-as", d.getGlobalId().toURL().toExternalForm()) - .add("sorg:name", d.getDisplayName()).add("type", "sorg:Dataset")); + job.add("object", rel); job.add("origin", Json.createObjectBuilder().add("id", SystemConfig.getDataverseSiteUrlStatic()) .add("inbox", SystemConfig.getDataverseSiteUrlStatic() + "/api/inbox").add("type", "Service")); job.add("target", target); - job.add("type", Json.createArrayBuilder().add("Announce").add("coar-notify:ReleaseAction")); + job.add("type", Json.createArrayBuilder().add("Announce").add("coar-notify:RelationshipAction")); HttpPost annPost = new HttpPost(); annPost.setURI(new URI(target.getString("inbox"))); String body = JsonUtil.prettyPrint(job.build()); - logger.fine("Body: " + body); + logger.info("Body: " + body); annPost.setEntity(new StringEntity(JsonUtil.prettyPrint(body), "utf-8")); annPost.setHeader("Content-Type", "application/ld+json"); return annPost; } - private String getBestPubId(JsonObject jo) { + private String getBestId(DatasetFieldType dft, JsonValue jv) { + //Primitive value + if(jv instanceof JsonString) { + return ((JsonString)jv).getString(); + } + //Compound - apply type specific logic to get best Id String id = null; - if (jo.containsKey(publicationURL.getLabel())) { - id = jo.getString(publicationURL.getLabel()); - } else if (jo.containsKey(publicationIDType.getLabel())) { - if ((jo.containsKey(publicationIDNumber.getLabel()))) { - String number = jo.getString(publicationIDNumber.getLabel()); + switch (dft.getName()) { + case RELATED_PUBLICATION: + JsonLDTerm publicationIDType = null; + JsonLDTerm publicationIDNumber = null; + JsonLDTerm publicationURL = null; + JsonObject jo = jv.asJsonObject(); + if (jo.containsKey(publicationURL.getLabel())) { + id = jo.getString(publicationURL.getLabel()); + } else if (jo.containsKey(publicationIDType.getLabel())) { + if ((jo.containsKey(publicationIDNumber.getLabel()))) { + String number = jo.getString(publicationIDNumber.getLabel()); - switch (jo.getString(publicationIDType.getLabel())) { - case "doi": - if (number.startsWith("https://doi.org/")) { - id = number; - } else if (number.startsWith("doi:")) { - id = "https://doi.org/" + number.substring(4); - } + switch (jo.getString(publicationIDType.getLabel())) { + case "doi": + if (number.startsWith("https://doi.org/")) { + id = number; + } else if (number.startsWith("doi:")) { + id = "https://doi.org/" + number.substring(4); + } - break; - case "DASH-URN": - if (number.startsWith("http")) { - id = number; + break; + case "DASH-URN": + if (number.startsWith("http")) { + id = number; + } + break; } - break; } } + break; + default: + break; } + return id; } From 63aeed1abe9c6155937d7f10076007be0ead2fae Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 11 Aug 2022 15:01:08 -0400 Subject: [PATCH 004/578] add workflow settings to main list per qa --- doc/sphinx-guides/source/installation/config.rst | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 2d294980720..3405285fefb 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -943,7 +943,7 @@ Some external tools are also ready to be translated, especially if they are usin Tools for Translators -+++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++ The list below depicts a set of tools that can be used to ease the amount of work necessary for translating the Dataverse software by facilitating this collaborative effort and enabling the reuse of previous work: @@ -2865,4 +2865,15 @@ For configuration details, see :ref:`mute-notifications`. :LDNMessageHosts ++++++++++++++++ -The comma-separated list of hosts allowed to send Dataverse Linked Data Notification messages. See :doc:`/api/linkeddatanotification` for details. ``*`` allows messages from anywhere (not recommended for production). By default, messages are not accepted from anywhere. +The comma-separated list of hosts allowed to send Dataverse Linked Data Notification messages. See :doc:`/api/linkeddatanotification` for details. ``*`` allows messages from anywhere (not recommended for production). By default, messages are not accepted from anywhere. + + +:LDN_TARGET ++++++++++++ + +The URL of an LDN Inbox to which the LDN Announce workflow step will send messages. See :doc:`/developers/workflows` for details. + +:LDNAnnounceRequiredFields +++++++++++++++++++++++++++ + +The list of parent dataset field names for which the LDN Announce workflow step should send messages. See :doc:`/developers/workflows` for details. From a4106590eece76b651d23d96b655409df96436c8 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 11 Aug 2022 15:54:59 -0400 Subject: [PATCH 005/578] start adding default bestID --- .../LDNAnnounceDatasetVersionStep.java | 34 +++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java index 1b373ffefa8..b2376ff89c3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -19,6 +19,7 @@ import java.net.URISyntaxException; import java.nio.charset.StandardCharsets; import java.util.Arrays; +import java.util.Collection; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; @@ -208,13 +209,30 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { return ((JsonString)jv).getString(); } //Compound - apply type specific logic to get best Id + JsonObject jo = jv.asJsonObject(); String id = null; switch (dft.getName()) { case RELATED_PUBLICATION: JsonLDTerm publicationIDType = null; JsonLDTerm publicationIDNumber = null; JsonLDTerm publicationURL = null; - JsonObject jo = jv.asJsonObject(); + + Collection childTypes = dft.getChildDatasetFieldTypes(); + for (DatasetFieldType cdft : childTypes) { + switch (cdft.getName()) { + case "publicationURL": + publicationURL = cdft.getJsonLDTerm(); + break; + case "publicationIDType": + publicationIDType = cdft.getJsonLDTerm(); + break; + case "publicationIDNumber": + publicationIDNumber = cdft.getJsonLDTerm(); + break; + } + } + + if (jo.containsKey(publicationURL.getLabel())) { id = jo.getString(publicationURL.getLabel()); } else if (jo.containsKey(publicationIDType.getLabel())) { @@ -227,8 +245,10 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { id = number; } else if (number.startsWith("doi:")) { id = "https://doi.org/" + number.substring(4); + } else { + //Assume a raw DOI, e.g. 10.5072/FK2ABCDEF + id = "https://doi.org/" + number; } - break; case "DASH-URN": if (number.startsWith("http")) { @@ -240,6 +260,16 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { } break; default: + Collection childDFTs = dft.getChildDatasetFieldTypes(); + for (DatasetFieldType cdft : childDFTs) { + String fieldname = cdft.getName(); + if(fieldname.contains("URL")) { + if(jo.containsKey(cdft.getJsonLDTerm().getLabel())) { + id=jo.getString(cdft.getJsonLDTerm().getLabel()); + break; + } + } + } break; } From 8b69c0f630ba26147e417339b12bdd99537833fc Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 11 Aug 2022 16:29:55 -0400 Subject: [PATCH 006/578] replace system.out.println call --- .../java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java index 127632bf711..f58dd0893c1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JSONLDUtil.java @@ -540,7 +540,7 @@ public static JsonObject decontextualizeJsonLD(String jsonLDString) { logger.fine("Decontextualized object: " + jsonld); return jsonld; } catch (JsonLdError e) { - System.out.println(e.getMessage()); + logger.warning(e.getMessage()); return null; } } From cbd20faf6e6ca8f74e68e48535ff6148a8680dad Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 11 Aug 2022 16:30:43 -0400 Subject: [PATCH 007/578] Improve default compound field handling and some reformatting --- .../LDNAnnounceDatasetVersionStep.java | 105 +++++++++++------- 1 file changed, 66 insertions(+), 39 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java index b2376ff89c3..ed19bd42a8e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -60,8 +60,6 @@ public class LDNAnnounceDatasetVersionStep implements WorkflowStep { private static final String LDN_TARGET = ":LDNTarget"; private static final String RELATED_PUBLICATION = "publication"; - - public LDNAnnounceDatasetVersionStep(Map paramSet) { new HashMap<>(paramSet); } @@ -76,7 +74,7 @@ public WorkflowStepResult run(WorkflowContext context) { CloseableHttpClient client = HttpClients.createDefault(); // build method - + HttpPost announcement; try { // First check that we have what is required @@ -84,14 +82,15 @@ public WorkflowStepResult run(WorkflowContext context) { DatasetVersion dv = d.getReleasedVersion(); List dvf = dv.getDatasetFields(); Map fields = new HashMap(); - List reqFields = Arrays.asList(((String) context.getSettings().getOrDefault(REQUIRED_FIELDS, "")).split(",\\s*")); + List reqFields = Arrays + .asList(((String) context.getSettings().getOrDefault(REQUIRED_FIELDS, "")).split(",\\s*")); for (DatasetField df : dvf) { - if(!df.isEmpty() && reqFields.contains(df.getDatasetFieldType().getName())) { + if (!df.isEmpty() && reqFields.contains(df.getDatasetFieldType().getName())) { fields.put(df.getDatasetFieldType().getName(), df); } } - //Loop through and send a message for each supported relationship - boolean success=false; + // Loop through and send a message for each supported relationship + boolean success = false; for (JsonObject rel : getObjects(context, fields).getValuesAs(JsonObject.class)) { announcement = buildAnnouncement(d, rel, target); // execute @@ -99,29 +98,30 @@ public WorkflowStepResult run(WorkflowContext context) { int code = response.getStatusLine().getStatusCode(); if (code >= 200 && code < 300) { // HTTP OK range - success=true; + success = true; logger.fine("Successfully sent message for " + rel.toString()); } else { String responseBody = new String(response.getEntity().getContent().readAllBytes(), StandardCharsets.UTF_8); ; - return new Failure((success ? "Partial failure":"") + "Error communicating with " + inboxUrl + " for relationship " + rel.toString() +". Server response: " + responseBody - + " (" + response + ")."); + return new Failure((success ? "Partial failure" : "") + "Error communicating with " + + inboxUrl + " for relationship " + rel.toString() + ". Server response: " + + responseBody + " (" + response + ")."); } } catch (Exception ex) { logger.log(Level.SEVERE, "Error communicating with remote server: " + ex.getMessage(), ex); - return new Failure((success ? "Partial failure":"") + "Error executing request: " + ex.getLocalizedMessage(), - "Cannot communicate with remote server."); + return new Failure((success ? "Partial failure" : "") + "Error executing request: " + + ex.getLocalizedMessage(), "Cannot communicate with remote server."); } } - //Any failure and we would have returned already. + // Any failure and we would have returned already. return OK; - } catch (URISyntaxException e) { - return new Failure("LDNAnnounceDatasetVersion workflow step failed: unable to parse inbox in :LDNTarget setting."); + return new Failure( + "LDNAnnounceDatasetVersion workflow step failed: unable to parse inbox in :LDNTarget setting."); } } return new Failure("LDNAnnounceDatasetVersion workflow step failed: :LDNTarget setting missing or invalid."); @@ -137,11 +137,14 @@ public void rollback(WorkflowContext context, Failure reason) { throw new UnsupportedOperationException("Not supported yet."); // This class does not need to resume. } - /**Scan through all fields and return an array of relationship JsonObjects with subjectId, relationship, objectId, and @context + /** + * Scan through all fields and return an array of relationship JsonObjects with + * subjectId, relationship, objectId, and @context * * @param ctxt * @param fields - * @return JsonArray of JsonObjects with subjectId, relationship, objectId, and @context + * @return JsonArray of JsonObjects with subjectId, relationship, objectId, + * and @context */ JsonArray getObjects(WorkflowContext ctxt, Map fields) { JsonArrayBuilder jab = Json.createArrayBuilder(); @@ -153,34 +156,33 @@ JsonArray getObjects(WorkflowContext ctxt, Map fields) { DatasetField field = entry.getValue(); DatasetFieldType dft = field.getDatasetFieldType(); JsonValue jv = OREMap.getJsonLDForField(field, false, emptyCvocMap, localContext); - //jv is a JsonArray for multi-val fields, so loop + // jv is a JsonArray for multi-val fields, so loop if (jv != null) { if (jv instanceof JsonArray) { JsonArray rels = (JsonArray) jv; Iterator iter = rels.iterator(); - while(iter.hasNext()) { - JsonValue jval =iter.next(); - jab.add(getRelationshipObject(dft, jval, d, localContext)); + while (iter.hasNext()) { + JsonValue jval = iter.next(); + jab.add(getRelationshipObject(dft, jval, d, localContext)); } } else { jab.add(getRelationshipObject(dft, jv, d, localContext)); } } - + } return jab.build(); } - - private JsonObject getRelationshipObject(DatasetFieldType dft, JsonValue jval, Dataset d, Map localContext) { + + private JsonObject getRelationshipObject(DatasetFieldType dft, JsonValue jval, Dataset d, + Map localContext) { String id = getBestId(dft, jval); return Json.createObjectBuilder().add("object", id).add("relationship", dft.getJsonLDTerm().getUrl()) .add("subject", d.getGlobalId().toURL().toString()).build(); } - HttpPost buildAnnouncement(Dataset d, JsonObject rel, JsonObject target) throws URISyntaxException { - JsonObjectBuilder job = Json.createObjectBuilder(); JsonArrayBuilder context = Json.createArrayBuilder().add("https://purl.org/coar/notify") .add("https://www.w3.org/ns/activitystreams"); @@ -204,11 +206,11 @@ HttpPost buildAnnouncement(Dataset d, JsonObject rel, JsonObject target) throws } private String getBestId(DatasetFieldType dft, JsonValue jv) { - //Primitive value - if(jv instanceof JsonString) { - return ((JsonString)jv).getString(); + // Primitive value + if (jv instanceof JsonString) { + return ((JsonString) jv).getString(); } - //Compound - apply type specific logic to get best Id + // Compound - apply type specific logic to get best Id JsonObject jo = jv.asJsonObject(); String id = null; switch (dft.getName()) { @@ -216,7 +218,7 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { JsonLDTerm publicationIDType = null; JsonLDTerm publicationIDNumber = null; JsonLDTerm publicationURL = null; - + Collection childTypes = dft.getChildDatasetFieldTypes(); for (DatasetFieldType cdft : childTypes) { switch (cdft.getName()) { @@ -231,8 +233,6 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { break; } } - - if (jo.containsKey(publicationURL.getLabel())) { id = jo.getString(publicationURL.getLabel()); } else if (jo.containsKey(publicationIDType.getLabel())) { @@ -246,7 +246,7 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { } else if (number.startsWith("doi:")) { id = "https://doi.org/" + number.substring(4); } else { - //Assume a raw DOI, e.g. 10.5072/FK2ABCDEF + // Assume a raw DOI, e.g. 10.5072/FK2ABCDEF id = "https://doi.org/" + number; } break; @@ -261,18 +261,45 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { break; default: Collection childDFTs = dft.getChildDatasetFieldTypes(); + // Loop through child fields and select one + // The order of preference is for a field with URL in the name, followed by one + // with 'ID',then 'Name', and as a last resort, a field. for (DatasetFieldType cdft : childDFTs) { String fieldname = cdft.getName(); - if(fieldname.contains("URL")) { - if(jo.containsKey(cdft.getJsonLDTerm().getLabel())) { - id=jo.getString(cdft.getJsonLDTerm().getLabel()); + if (fieldname.contains("URL")) { + if (jo.containsKey(cdft.getJsonLDTerm().getLabel())) { + id = jo.getString(cdft.getJsonLDTerm().getLabel()); break; } } } - break; - } + if (id == null) { + for (DatasetFieldType cdft : childDFTs) { + String fieldname = cdft.getName(); + if (fieldname.contains("ID") || fieldname.contains("Id")) { + if (jo.containsKey(cdft.getJsonLDTerm().getLabel())) { + id = jo.getString(cdft.getJsonLDTerm().getLabel()); + break; + } + + } + } + } + if (id == null) { + for (DatasetFieldType cdft : childDFTs) { + String fieldname = cdft.getName(); + + if (fieldname.contains("Name")) { + if (jo.containsKey(cdft.getJsonLDTerm().getLabel())) { + id = jo.getString(cdft.getJsonLDTerm().getLabel()); + break; + } + } + } + } + id = jo.getString(jo.keySet().iterator().next()); + } return id; } From 94d40e6c95a7786c671ac4c1abfca1eed21deb33 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 11 Aug 2022 19:37:46 -0400 Subject: [PATCH 008/578] add template custom instructions info --- .../source/user/dataverse-management.rst | 29 ++++++++++--------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/doc/sphinx-guides/source/user/dataverse-management.rst b/doc/sphinx-guides/source/user/dataverse-management.rst index efe98e8327c..ed90497da8c 100755 --- a/doc/sphinx-guides/source/user/dataverse-management.rst +++ b/doc/sphinx-guides/source/user/dataverse-management.rst @@ -44,7 +44,7 @@ To edit your Dataverse collection, navigate to your Dataverse collection's landi - :ref:`Theme `: upload a logo for your Dataverse collection, add a link to your department or personal website, add a custom footer image, and select colors for your Dataverse collection in order to brand it - :ref:`Widgets `: get code to add to your website to have your Dataverse collection display on it - :ref:`Permissions `: give other users permissions to your Dataverse collection, i.e.-can edit datasets, and see which users already have which permissions for your Dataverse collection -- :ref:`Dataset Templates `: these are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in +- :ref:`Dataset Templates `: these are useful when you want to provide custom instructions on how to fill out fields or have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in - :ref:`Dataset Guestbooks `: allows you to collect data about who is downloading the files from your datasets - :ref:`Featured Dataverse collections `: if you have one or more Dataverse collection, you can use this option to show them at the top of your Dataverse collection page to help others easily find interesting or important Dataverse collections - **Delete Dataverse**: you are able to delete your Dataverse collection as long as it is not published and does not have any draft datasets @@ -52,7 +52,7 @@ To edit your Dataverse collection, navigate to your Dataverse collection's landi .. _general-information: General Information ---------------------- +------------------- The General Information page is how you edit the information you filled in while creating your Dataverse collection. If you need to change or add a contact email address, this is the place to do it. Additionally, you can update the metadata elements used for datasets within the Dataverse collection, change which metadata fields are hidden, required, or optional, and update the facets you would like displayed for browsing the Dataverse collection. If you plan on using templates, you need to select the metadata fields on the General Information page. @@ -60,8 +60,8 @@ Tip: The metadata fields you select as required will appear on the Create Datase .. _theme: -Theme ---------- +Theme +----- The Theme features provides you with a way to customize the look of your Dataverse collection. You can: @@ -77,7 +77,7 @@ Supported image types for logo images and footer images are JPEG, TIFF, or PNG a .. _dataverse-widgets: Widgets --------------- +------- The Widgets feature provides you with code for you to put on your personal website to have your Dataverse collection displayed there. There are two types of Widgets for a Dataverse collection, a Dataverse collection Search Box widget and a Dataverse collection Listing widget. Once a Dataverse collection has been published, from the Widgets tab on the Dataverse collection's Theme + Widgets page, it is possible to copy the code snippets for the widget(s) you would like to add to your website. If you need to adjust the height of the widget on your website, you may do so by editing the `heightPx=500` parameter in the code snippet. @@ -94,7 +94,7 @@ The Dataverse Collection Listing Widget provides a listing of all your Dataverse .. _openscholar-dataverse-level: Adding Widgets to an OpenScholar Website -****************************************** +**************************************** #. Log in to your OpenScholar website #. Either build a new page or navigate to the page you would like to use to show the Dataverse collection widgets. #. Click on the Settings Cog and select Layout @@ -102,8 +102,8 @@ Adding Widgets to an OpenScholar Website .. _dataverse-permissions: -Roles & Permissions ---------------------- +Roles & Permissions +------------------- Dataverse installation user accounts can be granted roles that define which actions they are allowed to take on specific Dataverse collections, datasets, and/or files. Each role comes with a set of permissions, which define the specific actions that users may take. Roles and permissions may also be granted to groups. Groups can be defined as a collection of Dataverse installation user accounts, a collection of IP addresses (e.g. all users of a library's computers), or a collection of all users who log in using a particular institutional login (e.g. everyone who logs in with a particular university's account credentials). @@ -127,7 +127,7 @@ When you access a Dataverse collection's permissions page, you will see three se Please note that even on a newly created Dataverse collection, you may see user and groups have already been granted role(s) if your installation has ``:InheritParentRoleAssignments`` set. For more on this setting, see the :doc:`/installation/config` section of the Installation Guide. Setting Access Configurations -******************************* +***************************** Under the Permissions tab, you can click the "Edit Access" button to open a box where you can add to your Dataverse collection and what permissions are granted to those who add to your Dataverse collection. @@ -140,7 +140,7 @@ The second question on this page allows you to choose the role (and thus the per Both of these settings can be changed at any time. Assigning Roles to Users and Groups -************************************* +*********************************** Under the Users/Groups tab, you can add, edit, or remove the roles granted to users and groups on your Dataverse collection. A role is a set of permissions granted to a user or group when they're using your Dataverse collection. For example, giving your research assistant the "Contributor" role would give them the following self-explanatory permissions on your Dataverse collection and all datasets within your Dataverse collection: "ViewUnpublishedDataset", "DownloadFile", "EditDataset", and "DeleteDatasetDraft". They would, however, lack the "PublishDataset" permission, and thus would be unable to publish datasets on your Dataverse collection. If you wanted to give them that permission, you would give them a role with that permission, like the Curator role. Users and groups can hold multiple roles at the same time if needed. Roles can be removed at any time. All roles and their associated permissions are listed under the "Roles" tab of the same page. @@ -155,15 +155,16 @@ Note: If you need to assign a role to ALL user accounts in a Dataverse installat .. _dataset-templates: Dataset Templates -------------------- +----------------- -Templates are useful when you have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in, or if you want to use a custom set of Terms of Use and Access for multiple datasets in a Dataverse collection. In Dataverse Software 4.0+, templates are created at the Dataverse collection level, can be deleted (so it does not show for future datasets), set to default (not required), or can be copied so you do not have to start over when creating a new template with similar metadata from another template. When a template is deleted, it does not impact the datasets that have used the template already. +Templates are useful when you want to provide custom instructions on how to fill out a field, have several datasets that have the same information in multiple metadata fields that you would prefer not to have to keep manually typing in, or if you want to use a custom set of Terms of Use and Access for multiple datasets in a Dataverse collection. In Dataverse Software 4.0+, templates are created at the Dataverse collection level, can be deleted (so it does not show for future datasets), set to default (not required), or can be copied so you do not have to start over when creating a new template with similar metadata from another template. When a template is deleted, it does not impact the datasets that have used the template already. How do you create a template? #. Navigate to your Dataverse collection, click on the Edit Dataverse button and select Dataset Templates. #. Once you have clicked on Dataset Templates, you will be brought to the Dataset Templates page. On this page, you can 1) decide to use the dataset templates from your parent Dataverse collection 2) create a new dataset template or 3) do both. #. Click on the Create Dataset Template to get started. You will see that the template is the same as the create dataset page with an additional field at the top of the page to add a name for the template. +#. To add custom instructions, click on ''(None - click to add)'' and enter the instructions you wish users to see. If you wish to edit existing instructions, click on them to make the text editable. #. After adding information into the metadata fields you have information for and clicking Save and Add Terms, you will be brought to the page where you can add custom Terms of Use and Access. If you do not need custom Terms of Use and Access, click the Save Dataset Template, and only the metadata fields will be saved. #. After clicking Save Dataset Template, you will be brought back to the Manage Dataset Templates page and should see your template listed there now with the make default, edit, view, or delete options. #. A Dataverse collection does not have to have a default template and users can select which template they would like to use while on the Create Dataset page. @@ -174,7 +175,7 @@ How do you create a template? .. _dataset-guestbooks: Dataset Guestbooks ------------------------------ +------------------ Guestbooks allow you to collect data about who is downloading the files from your datasets. You can decide to collect account information (username, given name & last name, affiliation, etc.) as well as create custom questions (e.g., What do you plan to use this data for?). You are also able to download the data collected from the enabled guestbooks as CSV files to store and use outside of the Dataverse installation. @@ -227,7 +228,7 @@ Similarly to dataset linking, Dataverse collection linking allows a Dataverse co If you need to have a Dataverse collection linked to your Dataverse collection, please contact the support team for the Dataverse installation you are using. Publish Your Dataverse Collection -================================================================= +================================= Once your Dataverse collection is ready to go public, go to your Dataverse collection page, click on the "Publish" button on the right hand side of the page. A pop-up will appear to confirm that you are ready to actually Publish, since once a Dataverse collection From aa881820f537c283eb9b8caa999dddc63c8dc89f Mon Sep 17 00:00:00 2001 From: qqmyers Date: Fri, 12 Aug 2022 14:16:59 -0400 Subject: [PATCH 009/578] get blocks from metadataroot --- src/main/java/edu/harvard/iq/dataverse/Template.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/Template.java b/src/main/java/edu/harvard/iq/dataverse/Template.java index ba601c1df87..61f0a78656f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Template.java +++ b/src/main/java/edu/harvard/iq/dataverse/Template.java @@ -272,7 +272,7 @@ public void setMetadataValueBlocks() { Map instructionsMap = getInstructionsMap(); List viewMDB = new ArrayList<>(); - List editMDB=this.getDataverse().getMetadataBlocks(true); + List editMDB=this.getDataverse().getMetadataBlocks(false); //The metadatablocks in this template include any from the Dataverse it is associated with //plus any others where the template has a displayable field (i.e. from before a block was dropped in the dataverse/collection) From 62fe2143d61ed93d09c160961c315982b61e37f8 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 1 Sep 2022 14:55:03 -0400 Subject: [PATCH 010/578] fix parsing to match spec/dash msg --- .../java/edu/harvard/iq/dataverse/api/LDNInbox.java | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index fabc067abf7..5e434ec6bef 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -104,15 +104,13 @@ public Response acceptMessage(String body) { String objectKey = new JsonLDTerm(activityStreams, "object").getUrl(); if (jsonld.containsKey(objectKey)) { JsonObject msgObject = jsonld.getJsonObject(objectKey); - if (new JsonLDTerm(activityStreams, "Relationship").getUrl().equals(msgObject.getJsonString("@type"))) { + if (new JsonLDTerm(activityStreams, "Relationship").getUrl().equals(msgObject.getJsonString("type"))) { // We have a relationship message - need to get the subject and object and // relationship type - String subjectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "subject").getUrl()) - .getString("@id"); - String objectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "object").getUrl()) - .getString("@id"); + String subjectId = msgObject.getString((new JsonLDTerm(activityStreams, "subject").getUrl()); + String objectId = msgObject.getString(new JsonLDTerm(activityStreams, "object").getUrl()); String relationshipId = msgObject - .getJsonObject(new JsonLDTerm(activityStreams, "relationship").getUrl()).getString("@id"); + .getString(new JsonLDTerm(activityStreams, "relationship").getUrl()); if (subjectId != null && objectId != null && relationshipId != null) { // Strip the URL part from a relationship ID/URL assuming a usable label exists // after a # char. Default is to use the whole URI. From e5228af91781be53ba69e1ad803d03130383d026 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 1 Sep 2022 16:04:31 -0400 Subject: [PATCH 011/578] use getString for 'Relationship', revert other changes --- .../java/edu/harvard/iq/dataverse/api/LDNInbox.java | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index 5e434ec6bef..96f15dd5c81 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -16,6 +16,7 @@ import edu.harvard.iq.dataverse.util.json.JSONLDUtil; import edu.harvard.iq.dataverse.util.json.JsonLDNamespace; import edu.harvard.iq.dataverse.util.json.JsonLDTerm; +import edu.harvard.iq.dataverse.util.json.JsonUtil; import java.util.Date; import java.util.Map; @@ -96,6 +97,8 @@ public Response acceptMessage(String body) { if (jsonld == null) { throw new BadRequestException("Could not parse message to find acceptable citation link to a dataset."); } + //To Do - lower level for PR + logger.info(JsonUtil.prettyPrint(jsonld)); String relationship = "isRelatedTo"; String name = null; JsonLDNamespace activityStreams = JsonLDNamespace.defineNamespace("as", @@ -104,13 +107,13 @@ public Response acceptMessage(String body) { String objectKey = new JsonLDTerm(activityStreams, "object").getUrl(); if (jsonld.containsKey(objectKey)) { JsonObject msgObject = jsonld.getJsonObject(objectKey); - if (new JsonLDTerm(activityStreams, "Relationship").getUrl().equals(msgObject.getJsonString("type"))) { + if (new JsonLDTerm(activityStreams, "Relationship").getUrl().equals(msgObject.getString("@type"))) { // We have a relationship message - need to get the subject and object and // relationship type - String subjectId = msgObject.getString((new JsonLDTerm(activityStreams, "subject").getUrl()); - String objectId = msgObject.getString(new JsonLDTerm(activityStreams, "object").getUrl()); + String subjectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "subject").getUrl()).getString("@id"); + String objectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "object").getUrl()).getString("@id"); String relationshipId = msgObject - .getString(new JsonLDTerm(activityStreams, "relationship").getUrl()); + .getJsonObject(new JsonLDTerm(activityStreams, "relationship").getUrl()).getString("@id"); if (subjectId != null && objectId != null && relationshipId != null) { // Strip the URL part from a relationship ID/URL assuming a usable label exists // after a # char. Default is to use the whole URI. From fb56d0413bda84901432310364f2a96093829739 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 1 Sep 2022 16:04:51 -0400 Subject: [PATCH 012/578] temporarily drop name/type in display --- src/main/webapp/dataverseuser.xhtml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/main/webapp/dataverseuser.xhtml b/src/main/webapp/dataverseuser.xhtml index 9fb6c3bdac0..59b02744a56 100644 --- a/src/main/webapp/dataverseuser.xhtml +++ b/src/main/webapp/dataverseuser.xhtml @@ -406,9 +406,11 @@ - + + + + - #{item.theObject.getDisplayName()} From 8ecb8e46f1a3d0910176a6f3dbcd06eea62cc853 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 1 Sep 2022 16:13:57 -0400 Subject: [PATCH 013/578] misplaced } --- src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index 96f15dd5c81..b69a22d5630 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -163,11 +163,11 @@ public Response acceptMessage(String body) { } } else { - } + // This isn't a Relationship announcement message - ignore logger.info("This is not a relationship announcement - ignoring message of type " + msgObject.getJsonString("@type")); - + } } if (!sent) { From 789b3c91211ef1561792bd76846c3bd6242c8b76 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Tue, 6 Sep 2022 15:42:40 -0400 Subject: [PATCH 014/578] Add callback loop and make display tolerant wrt name/type being found --- .../harvard/iq/dataverse/api/LDNInbox.java | 63 +++++++++++++++++-- src/main/java/propertyFiles/Bundle.properties | 1 + src/main/webapp/dataverseuser.xhtml | 6 +- 3 files changed, 62 insertions(+), 8 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index b69a22d5630..32bb1d12b36 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -23,12 +23,15 @@ import java.util.Optional; import java.util.Set; import java.io.StringWriter; +import java.net.URI; +import java.net.URL; import java.sql.Timestamp; import java.util.logging.Logger; import javax.ejb.EJB; import javax.json.Json; import javax.json.JsonObject; +import javax.json.JsonObjectBuilder; import javax.json.JsonValue; import javax.json.JsonWriter; import javax.servlet.http.HttpServletRequest; @@ -41,6 +44,13 @@ import javax.ws.rs.core.Context; import javax.ws.rs.core.Response; +import org.apache.commons.httpclient.HttpClient; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpGet; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.util.EntityUtils; + @Path("inbox") public class LDNInbox extends AbstractApiBean { @@ -99,8 +109,9 @@ public Response acceptMessage(String body) { } //To Do - lower level for PR logger.info(JsonUtil.prettyPrint(jsonld)); - String relationship = "isRelatedTo"; + //String relationship = "isRelatedTo"; String name = null; + String itemType = null; JsonLDNamespace activityStreams = JsonLDNamespace.defineNamespace("as", "https://www.w3.org/ns/activitystreams#"); JsonLDNamespace ietf = JsonLDNamespace.defineNamespace("ietf", "http://www.iana.org/assignments/relation/"); @@ -112,25 +123,69 @@ public Response acceptMessage(String body) { // relationship type String subjectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "subject").getUrl()).getString("@id"); String objectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "object").getUrl()).getString("@id"); + // Best-effort to get name by following redirects and looing for a 'name' field in the returned json + try (CloseableHttpClient client = HttpClients.createDefault()) { + logger.fine("Getting " + subjectId); + HttpGet get = new HttpGet(new URI(subjectId)); + get.addHeader("Accept", "application/json"); + + int statusCode=0; + do { + CloseableHttpResponse response = client.execute(get); + statusCode = response.getStatusLine().getStatusCode(); + switch (statusCode) { + case 302: + case 303: + String location=response.getFirstHeader("location").getValue(); + logger.fine("Redirecting to: " + location); + get = new HttpGet(location); + get.addHeader("Accept", "application/json"); + + break; + case 200: + String responseString = EntityUtils.toString(response.getEntity(), "UTF-8"); + logger.fine("Received: " + responseString); + JsonObject job = JsonUtil.getJsonObject(responseString); + name = job.getString("name", null); + itemType = job.getString("type", null); + break; + default: + logger.fine("Received " + statusCode + " when accessing " + objectId); + } + } while(statusCode == 302); + } catch (Exception e) { + logger.fine("Unable to get name from " + objectId); + logger.fine(e.getLocalizedMessage()); + } String relationshipId = msgObject .getJsonObject(new JsonLDTerm(activityStreams, "relationship").getUrl()).getString("@id"); if (subjectId != null && objectId != null && relationshipId != null) { // Strip the URL part from a relationship ID/URL assuming a usable label exists // after a # char. Default is to use the whole URI. - relationship = relationshipId.substring(relationship.indexOf("#") + 1); + int index = relationshipId.indexOf("#"); + logger.fine("Found # at " + index + " in " + relationshipId); + String relationship = relationshipId.substring(index + 1); // Parse the URI as a PID and see if this Dataverse instance has this dataset String pid = GlobalId.getInternalFormOfPID(objectId); Optional id = GlobalId.parse(pid); if (id.isPresent()) { Dataset dataset = datasetSvc.findByGlobalId(pid); if (dataset != null) { - JsonObject citingResource = Json.createObjectBuilder().add("@id", subjectId) - .add("relationship", relationship).build(); + JsonObjectBuilder citingResourceBuilder = Json.createObjectBuilder().add("@id", subjectId) + .add("relationship", relationship); + if(name!=null && !name.isBlank()) { + citingResourceBuilder.add("name",name); + } + if(itemType!=null && !itemType.isBlank()) { + citingResourceBuilder.add("@type",itemType.substring(0,1).toUpperCase() + itemType.substring(1)); + } + JsonObject citingResource = citingResourceBuilder.build(); StringWriter sw = new StringWriter(128); try (JsonWriter jw = Json.createWriter(sw)) { jw.write(citingResource); } String jsonstring = sw.toString(); + logger.fine("Storing: " + jsonstring); Set ras = roleService.rolesAssignments(dataset); roleService.rolesAssignments(dataset).stream() diff --git a/src/main/java/propertyFiles/Bundle.properties b/src/main/java/propertyFiles/Bundle.properties index 5b6216aaff1..74d6cbc3d60 100644 --- a/src/main/java/propertyFiles/Bundle.properties +++ b/src/main/java/propertyFiles/Bundle.properties @@ -217,6 +217,7 @@ notification.workflowFailed=An external workflow run on {0} in {1} has failed. C notification.workflowSucceeded=An external workflow run on {0} in {1} has succeeded. Check your email and/or view the Dataset page which may have additional details. notification.statusUpdated=The status of dataset {0} has been updated to {1}. notification.datasetMentioned=Announcement Received: Newly released {0} {2} {3} Dataset {4}. +notification.datasetMentioned.itemType=Resource notification.ingestCompleted=Dataset {1} has one or more tabular files that completed the tabular ingest process and are available in archival formats. notification.ingestCompletedWithErrors=Dataset {1} has one or more tabular files that are available but are not supported for tabular ingest. diff --git a/src/main/webapp/dataverseuser.xhtml b/src/main/webapp/dataverseuser.xhtml index 59b02744a56..5538072b7be 100644 --- a/src/main/webapp/dataverseuser.xhtml +++ b/src/main/webapp/dataverseuser.xhtml @@ -406,11 +406,9 @@ - - - - + + #{item.theObject.getDisplayName()} From 37ebf7dd04066e772b2ef2819abfb961860fd38a Mon Sep 17 00:00:00 2001 From: qqmyers Date: Fri, 23 Sep 2022 10:24:09 -0400 Subject: [PATCH 015/578] add fields missing in https://notify.coar-repositories.org/scenarios/10/ --- .../workflow/internalspi/LDNAnnounceDatasetVersionStep.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java index ed19bd42a8e..2aea94d08e5 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -178,7 +178,7 @@ private JsonObject getRelationshipObject(DatasetFieldType dft, JsonValue jval, D Map localContext) { String id = getBestId(dft, jval); return Json.createObjectBuilder().add("object", id).add("relationship", dft.getJsonLDTerm().getUrl()) - .add("subject", d.getGlobalId().toURL().toString()).build(); + .add("subject", d.getGlobalId().toURL().toString()).add("id", "urn:uuid:" + UUID.randomUUID().toString()).add("type","Relationship").build(); } HttpPost buildAnnouncement(Dataset d, JsonObject rel, JsonObject target) throws URISyntaxException { From ceb3ff576efc6da0ba50da43c2c2eaac10a8f172 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Wed, 26 Oct 2022 16:12:31 -0400 Subject: [PATCH 016/578] debugging --- src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index 32bb1d12b36..c916cff9179 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -80,6 +80,7 @@ public class LDNInbox extends AbstractApiBean { @Path("/") @Consumes("application/ld+json, application/json-ld") public Response acceptMessage(String body) { + try { IpAddress origin = new DataverseRequest(null, httpRequest).getSourceAddress(); String whitelist = settingsService.get(SettingsServiceBean.Key.LDNMessageHosts.toString(), ""); // Only do something if we listen to this host @@ -232,6 +233,12 @@ public Response acceptMessage(String body) { logger.info("Ignoring message from IP address: " + origin.toString()); throw new ForbiddenException("Inbox does not acept messages from this address"); } + } catch (Throwable t) { + logger.severe(t.getLocalizedMessage()); + t.printStackTrace(); + + throw t; + } return ok("Message Received"); } } From 62888e26fdecae70c55e5eab9e0f72683ef3af54 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Thu, 27 Oct 2022 12:09:39 -0400 Subject: [PATCH 017/578] added logging/switched to info for HDC debugging --- .../edu/harvard/iq/dataverse/api/LDNInbox.java | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index c916cff9179..4cf272c1d88 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -126,7 +126,7 @@ public Response acceptMessage(String body) { String objectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "object").getUrl()).getString("@id"); // Best-effort to get name by following redirects and looing for a 'name' field in the returned json try (CloseableHttpClient client = HttpClients.createDefault()) { - logger.fine("Getting " + subjectId); + logger.info("Getting " + subjectId); HttpGet get = new HttpGet(new URI(subjectId)); get.addHeader("Accept", "application/json"); @@ -138,25 +138,25 @@ public Response acceptMessage(String body) { case 302: case 303: String location=response.getFirstHeader("location").getValue(); - logger.fine("Redirecting to: " + location); + logger.info("Redirecting to: " + location); get = new HttpGet(location); get.addHeader("Accept", "application/json"); break; case 200: String responseString = EntityUtils.toString(response.getEntity(), "UTF-8"); - logger.fine("Received: " + responseString); + logger.info("Received: " + responseString); JsonObject job = JsonUtil.getJsonObject(responseString); name = job.getString("name", null); itemType = job.getString("type", null); break; default: - logger.fine("Received " + statusCode + " when accessing " + objectId); + logger.info("Received " + statusCode + " when accessing " + subjectId); } } while(statusCode == 302); } catch (Exception e) { - logger.fine("Unable to get name from " + objectId); - logger.fine(e.getLocalizedMessage()); + logger.info("Unable to get name from " + subjectId); + logger.info(e.getLocalizedMessage()); } String relationshipId = msgObject .getJsonObject(new JsonLDTerm(activityStreams, "relationship").getUrl()).getString("@id"); @@ -164,7 +164,7 @@ public Response acceptMessage(String body) { // Strip the URL part from a relationship ID/URL assuming a usable label exists // after a # char. Default is to use the whole URI. int index = relationshipId.indexOf("#"); - logger.fine("Found # at " + index + " in " + relationshipId); + logger.info("Found # at " + index + " in " + relationshipId); String relationship = relationshipId.substring(index + 1); // Parse the URI as a PID and see if this Dataverse instance has this dataset String pid = GlobalId.getInternalFormOfPID(objectId); @@ -186,7 +186,7 @@ public Response acceptMessage(String body) { jw.write(citingResource); } String jsonstring = sw.toString(); - logger.fine("Storing: " + jsonstring); + logger.info("Storing: " + jsonstring); Set ras = roleService.rolesAssignments(dataset); roleService.rolesAssignments(dataset).stream() @@ -209,6 +209,7 @@ public Response acceptMessage(String body) { sent = true; } } else { + logger.info("Didn't find dataset"); // We don't have a dataset corresponding to the object of the relationship - do // nothing } From d714a1a2b68ff77eaa01f07422bcd64c95595c4b Mon Sep 17 00:00:00 2001 From: qqmyers Date: Tue, 23 Apr 2024 17:15:29 -0400 Subject: [PATCH 018/578] merge issues --- .../workflow/internalspi/LDNAnnounceDatasetVersionStep.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java index 9d29cba42ff..abe9f26f98d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -35,6 +35,7 @@ import jakarta.json.JsonArrayBuilder; import jakarta.json.JsonObject; import jakarta.json.JsonObjectBuilder; +import jakarta.json.JsonString; import jakarta.json.JsonValue; import org.apache.http.client.methods.CloseableHttpResponse; @@ -177,7 +178,7 @@ private JsonObject getRelationshipObject(DatasetFieldType dft, JsonValue jval, D Map localContext) { String id = getBestId(dft, jval); return Json.createObjectBuilder().add("object", id).add("relationship", dft.getJsonLDTerm().getUrl()) - .add("subject", d.getGlobalId().toURL().toString()).add("id", "urn:uuid:" + UUID.randomUUID().toString()).add("type","Relationship").build(); + .add("subject", d.getGlobalId().asURL().toString()).add("id", "urn:uuid:" + UUID.randomUUID().toString()).add("type","Relationship").build(); } HttpPost buildAnnouncement(Dataset d, JsonObject rel, JsonObject target) throws URISyntaxException { From 5ee7308623ce19ceecb5853836b0d9b253ce3d07 Mon Sep 17 00:00:00 2001 From: Don Sizemore Date: Fri, 24 May 2024 12:02:48 -0400 Subject: [PATCH 019/578] #8856 last minute updates, two years later --- .../source/installation/prerequisites.rst | 10 +++++----- doc/sphinx-guides/source/installation/shibboleth.rst | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index 63503a2f4f0..16937f1b1b8 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -14,7 +14,7 @@ After following all the steps below, you can proceed to the :doc:`installation-m Linux ----- -We assume you plan to run your Dataverse installation on Linux and we recommend RHEL or a derivative such as RockyLinux or AlmaLinux, which is the distribution family tested by the Dataverse Project team. These instructions are written for RHEL9 and derivatives with notes for RHEL 8 and earlier, but a number of community members have successfully installed Dataverse in Debian and Ubuntu environments. +We assume you plan to run your Dataverse installation on Linux and we recommend RHEL or a derivative such as RockyLinux, which is the distribution family tested by the Dataverse Project team. These instructions are written for RHEL9 and derivatives with notes for RHEL 8, but Dataverse is known to work well in Debian, Ubuntu, and most any modern Linux distribution. Java ---- @@ -298,7 +298,7 @@ On a Red Hat or derivative Linux distribution, you can install ImageMagick with # dnf install ImageMagick (most RedHat systems will have it pre-installed). -When installed using standard ``yum`` mechanism, above, the executable for the ImageMagick convert utility will be located at ``/usr/bin/convert``. No further configuration steps will then be required. +When installed using standard ``dnf`` mechanism, above, the executable for the ImageMagick convert utility will be located at ``/usr/bin/convert``. No further configuration steps will then be required. If the installed location of the convert executable is different from ``/usr/bin/convert``, you will also need to specify it in your Payara configuration using the JVM option, below. For example:: @@ -341,7 +341,7 @@ RHEL 8 users will need to enable the CodeReady-Builder repository:: subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms -Rocky or AlmaLinux 8.3+ users will need to enable the PowerTools repository:: +Rocky 8 users will need to enable the PowerTools repository:: dnf config-manager --enable powertools @@ -472,11 +472,11 @@ The following commands are intended to be run as root but we are aware that Pyth Install Python 3.9:: - dnf install python39 + dnf install python3 Install Counter Processor Python requirements:: - python3.9 -m ensurepip + python3 -m ensurepip cd /usr/local/counter-processor-0.1.04 pip3 install -r requirements.txt diff --git a/doc/sphinx-guides/source/installation/shibboleth.rst b/doc/sphinx-guides/source/installation/shibboleth.rst index 5195b20bb26..70a1cdc2b83 100644 --- a/doc/sphinx-guides/source/installation/shibboleth.rst +++ b/doc/sphinx-guides/source/installation/shibboleth.rst @@ -39,14 +39,14 @@ Install Shibboleth Installing Shibboleth will give us both the ``shibd`` service and the ``mod_shib`` Apache module. -Install Shibboleth Yum Repo +Install Shibboleth DNF Repo ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The Shibboleth project now provides `a web form `_ to generate an appropriate package repository for use with YUM/DNF. +The Shibboleth project now provides `a web form `_ to generate an appropriate package repository for use with DNF. You'll want to copy-paste the form results into ``/etc/yum.repos.d/shibboleth.repo`` or wherever is most appropriate for your operating system. -Install Shibboleth Via Yum +Install Shibboleth Via DNF ^^^^^^^^^^^^^^^^^^^^^^^^^^ Please note that during the installation it's ok to import GPG keys from the Shibboleth project. We trust them. From 8aa7d693ccdc56b5b68271d85bf1201fd1a5d94d Mon Sep 17 00:00:00 2001 From: Don Sizemore Date: Fri, 7 Jun 2024 14:44:04 -0400 Subject: [PATCH 020/578] #8856 update rserve-setup.sh for RHEL/Rocky 9 --- scripts/r/rserve/rserve-setup.sh | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/scripts/r/rserve/rserve-setup.sh b/scripts/r/rserve/rserve-setup.sh index 48ee747499a..c8f5b630eae 100755 --- a/scripts/r/rserve/rserve-setup.sh +++ b/scripts/r/rserve/rserve-setup.sh @@ -39,17 +39,14 @@ else echo "Rserve password file (/etc/Rserv.pwd) already exists." fi -if [ ! -f /etc/init.d/rserve ] +if [ ! -f /usr/lib/systemd/system/rserve.service ] then - echo "Installing Rserve startup file." - install rserve-startup.sh /etc/init.d/rserve - chkconfig rserve on - echo "You can start Rserve daemon by executing" - echo " service rserve start" - echo - echo "If this is a RedHat/CentOS 7/8 system, you may want to use the systemctl file rserve.service instead (provided in this directory)" + echo "Installing Rserve systemd unit file." + cp rserve.service /usr/lib/systemd/system + systemctl start rserve + systemctl enable rserve else - echo "Rserve startup file already in place." + echo "Rserve systemd unit file already in place." fi if [ ! -d /var/run/rserve ] From 7086afa2d49f9d9d0df35b3361e43dcbe6b13226 Mon Sep 17 00:00:00 2001 From: Don Sizemore Date: Fri, 7 Jun 2024 15:28:15 -0400 Subject: [PATCH 021/578] #8856 document R module compilation requirements --- doc/sphinx-guides/source/installation/prerequisites.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index 16937f1b1b8..3f4aa55e0f6 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -345,6 +345,10 @@ Rocky 8 users will need to enable the PowerTools repository:: dnf config-manager --enable powertools +To compile the modules below, on a RHEL/Rocky-based system you'll minimally need the following packages: + + dnf install openssl-devel libcurl-devel + Finally, install R with :fixedwidthplain:`dnf`:: dnf install R-core R-core-devel From 5d36cfa4219fc709d39a3bdc63abda85234a774e Mon Sep 17 00:00:00 2001 From: Don Sizemore Date: Tue, 11 Jun 2024 14:24:22 -0400 Subject: [PATCH 022/578] Update doc/sphinx-guides/source/installation/prerequisites.rst Co-authored-by: Philip Durbin --- doc/sphinx-guides/source/installation/prerequisites.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index 3f4aa55e0f6..55b1a5ea828 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -14,7 +14,7 @@ After following all the steps below, you can proceed to the :doc:`installation-m Linux ----- -We assume you plan to run your Dataverse installation on Linux and we recommend RHEL or a derivative such as RockyLinux, which is the distribution family tested by the Dataverse Project team. These instructions are written for RHEL9 and derivatives with notes for RHEL 8, but Dataverse is known to work well in Debian, Ubuntu, and most any modern Linux distribution. +We assume you plan to run your Dataverse installation on Linux and we recommend RHEL or a derivative such as Rocky Linux, which is the distribution family tested by the Dataverse Project team. These instructions are written for RHEL9 and derivatives with notes for RHEL 8, but Dataverse is known to work well in Debian, Ubuntu, and most any modern Linux distribution. Java ---- From 9dd4d1a038db26c86f580e07169897c2c14073c2 Mon Sep 17 00:00:00 2001 From: Don Sizemore Date: Tue, 11 Jun 2024 14:24:56 -0400 Subject: [PATCH 023/578] Update doc/sphinx-guides/source/installation/prerequisites.rst Co-authored-by: Philip Durbin --- doc/sphinx-guides/source/installation/prerequisites.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index 55b1a5ea828..25f8eb5b20a 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -104,7 +104,7 @@ You are welcome to experiment with newer versions of PostgreSQL, but please note Installing PostgreSQL ===================== -The application is currently tested on PostgreSQL version 13, though versions 14-16 are supported as of v6.2. We recommend installing the latest version that is available for your OS distribution. *For example*, to install PostgreSQL 13 under RHEL9/derivative:: +The application is currently tested on PostgreSQL version 13, though versions 14-16 are supported as of v6.2. To install PostgreSQL 13 under RHEL9/derivative:: # sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm # sudo dnf check-update From 0440ab91dbd0aa859b1422c7bea5ab31d95626a0 Mon Sep 17 00:00:00 2001 From: Don Sizemore Date: Tue, 16 Jul 2024 09:49:24 -0400 Subject: [PATCH 024/578] Update doc/sphinx-guides/source/installation/prerequisites.rst Co-authored-by: Ben Companjen --- doc/sphinx-guides/source/installation/prerequisites.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/installation/prerequisites.rst b/doc/sphinx-guides/source/installation/prerequisites.rst index 25f8eb5b20a..7859cdd06fd 100644 --- a/doc/sphinx-guides/source/installation/prerequisites.rst +++ b/doc/sphinx-guides/source/installation/prerequisites.rst @@ -113,7 +113,7 @@ The application is currently tested on PostgreSQL version 13, though versions 14 # sudo /usr/bin/systemctl start postgresql-13 # sudo /usr/bin/systemctl enable postgresql-13 -For RHEL8/derivative the process would be identical but requires a disabling the OS' built-in postgresql module with ``dnf -qy module disable postgresql``. +For RHEL8/derivative the process would be identical but requires a disabling the OS's built-in postgresql module with ``dnf -qy module disable postgresql``. Configuring Database Access for the Dataverse Installation (and the Dataverse Software Installer) ================================================================================================= From a1b5f34da88f13e57fdb75ee0fed63c5f20c05e4 Mon Sep 17 00:00:00 2001 From: Leonid Andreev Date: Mon, 12 May 2025 13:59:14 -0400 Subject: [PATCH 025/578] a quick implementation of a throttling-down setting for the harvesting client calls (the only thing I want to add is an option of enabling this setting for specific clients; similarly to how ingest size limits can be for all, or some specific formats only. #11473 --- .../harvest/client/HarvesterServiceBean.java | 18 ++++++++++++++++++ .../settings/SettingsServiceBean.java | 4 +++- .../iq/dataverse/util/SystemConfig.java | 15 +++++++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java index d7830991cff..dcdb382d8ac 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java @@ -44,6 +44,7 @@ import edu.harvard.iq.dataverse.harvest.client.oai.OaiHandler; import edu.harvard.iq.dataverse.harvest.client.oai.OaiHandlerException; import edu.harvard.iq.dataverse.search.IndexServiceBean; +import edu.harvard.iq.dataverse.util.SystemConfig; import java.io.FileOutputStream; import java.io.FileWriter; import java.io.InputStream; @@ -84,6 +85,8 @@ public class HarvesterServiceBean { EjbDataverseEngine engineService; @EJB IndexServiceBean indexService; + @EJB + SystemConfig systemConfig; private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean"); private static final SimpleDateFormat logFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH-mm-ss"); @@ -258,6 +261,14 @@ private void harvestOAI(DataverseRequest dataverseRequest, HarvestingClient harv } private void harvestOAIviaListIdentifiers(OaiHandler oaiHandler, DataverseRequest dataverseRequest, HarvestingClient harvestingClient, HttpClient httpClient, List failedIdentifiers, List deletedIdentifiers, List harvestedDatasetIds, Logger harvesterLogger, PrintWriter importCleanupLog) throws OaiHandlerException, StopHarvestException { + int sleepBetweenCalls = 0; + Float clientRequestInterval = systemConfig.getHarvestingClientRequestInterval(); + if (clientRequestInterval != null) { + sleepBetweenCalls = (int)(clientRequestInterval * 1000); + } + + logger.info("Sleep interval in milliseconds: "+sleepBetweenCalls); + for (Iterator
idIter = oaiHandler.runListIdentifiers(); idIter.hasNext();) { // Before each iteration, check if this harvesting job needs to be aborted: if (checkIfStoppingJob(harvestingClient)) { @@ -279,6 +290,13 @@ private void harvestOAIviaListIdentifiers(OaiHandler oaiHandler, DataverseReques MutableBoolean getRecordErrorOccurred = new MutableBoolean(false); + if (sleepBetweenCalls > 0) { + try { + Thread.sleep(sleepBetweenCalls); + } catch (InterruptedException iex) { + logger.warning("InterruptedException trying to sleep for " + sleepBetweenCalls + " milliseconds"); + } + } // Retrieve and process this record with a separate GetRecord call: Long datasetId = processRecord(dataverseRequest, harvesterLogger, importCleanupLog, oaiHandler, identifier, getRecordErrorOccurred, deletedIdentifiers, dateStamp, httpClient); diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java index 5b0a178969b..1e9d8590678 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java @@ -686,7 +686,9 @@ Whether Harvesting (OAI) service is enabled */ StoreIngestedTabularFilesWithVarHeaders, - ContactFeedbackMessageSizeLimit + ContactFeedbackMessageSizeLimit, + + HarvestingClientCallRateLimit ; @Override diff --git a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java index e4a62bf383a..9008c3f7acc 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java @@ -512,6 +512,21 @@ public long getTabularIngestSizeLimit() { return -1; } + public Float getHarvestingClientRequestInterval() { + String limitEntry = settingsService.getValueForKey(SettingsServiceBean.Key.HarvestingClientCallRateLimit); + + if (limitEntry != null) { + try { + Float sleepInterval = Float.valueOf(limitEntry); + return sleepInterval; + } catch (NumberFormatException nfe) { + logger.warning("Invalid value for HarvestingClientCallRateLimit option? - " + limitEntry); + } + } + + return null; + } + public long getTabularIngestSizeLimit(String formatName) { // This method returns the size limit set specifically for this format name, // if available, otherwise - the blanket limit that applies to all tabular From 723a5e5555bb41e7115283972b03e6750c1b8dda Mon Sep 17 00:00:00 2001 From: qqmyers Date: Wed, 20 Aug 2025 16:41:50 -0400 Subject: [PATCH 026/578] Handle Anubis forwarding as http via "X-Forwarded-Proto" assuming Apache/Nginx can add that. --- .../dataverse/api/auth/SignedUrlAuthMechanism.java | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java index 30e8a3b9ca4..b6057ad875b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java @@ -72,6 +72,18 @@ private User getAuthenticatedUserFromSignedUrl(ContainerRequestContext container } if (targetUser != null && userApiToken != null) { String signedUrl = URLDecoder.decode(uriInfo.getRequestUri().toString(), StandardCharsets.UTF_8); + + System.out.println("Orig URL: " + containerRequestContext.getUriInfo().getRequestUri().toString()); + String forwardedProto = containerRequestContext.getHeaderString("X-Forwarded-Proto"); + System.out.println("XFP: " + forwardedProto); + + + if (forwardedProto != null && !forwardedProto.isEmpty()) { + if ("https".equalsIgnoreCase(forwardedProto) && signedUrl.startsWith("http:")) { + signedUrl = "https" + signedUrl.substring(4); + } + } + String requestMethod = containerRequestContext.getMethod(); String signedUrlSigningKey = JvmSettings.API_SIGNING_SECRET.lookupOptional().orElse("") + userApiToken.getTokenString(); boolean isSignedUrlValid = UrlSignerUtil.isValidUrl(signedUrl, userId, requestMethod, signedUrlSigningKey); From 8c3a72e332f3bf7b6dbfac3b7fd8315b2d7a024b Mon Sep 17 00:00:00 2001 From: Jim Myers Date: Tue, 2 Sep 2025 15:28:23 -0400 Subject: [PATCH 027/578] use logger.fine --- .../iq/dataverse/api/auth/SignedUrlAuthMechanism.java | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java index b6057ad875b..d8457304e18 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java @@ -16,6 +16,7 @@ import java.net.URLDecoder; import java.nio.charset.StandardCharsets; +import java.util.logging.Logger; import static edu.harvard.iq.dataverse.util.UrlSignerUtil.SIGNED_URL_TOKEN; import static edu.harvard.iq.dataverse.util.UrlSignerUtil.SIGNED_URL_USER; @@ -33,6 +34,8 @@ public class SignedUrlAuthMechanism implements AuthMechanism { @Inject protected PrivateUrlServiceBean privateUrlSvc; + private static final Logger logger = Logger.getLogger(SignedUrlAuthMechanism.class.getCanonicalName()); + @Override public User findUserFromRequest(ContainerRequestContext containerRequestContext) throws WrappedAuthErrorResponse { String signedUrlRequestParameter = getSignedUrlRequestParameter(containerRequestContext); @@ -73,9 +76,9 @@ private User getAuthenticatedUserFromSignedUrl(ContainerRequestContext container if (targetUser != null && userApiToken != null) { String signedUrl = URLDecoder.decode(uriInfo.getRequestUri().toString(), StandardCharsets.UTF_8); - System.out.println("Orig URL: " + containerRequestContext.getUriInfo().getRequestUri().toString()); + logger.fine("Original URL: " + containerRequestContext.getUriInfo().getRequestUri().toString()); String forwardedProto = containerRequestContext.getHeaderString("X-Forwarded-Proto"); - System.out.println("XFP: " + forwardedProto); + logger.fine("X-Forwarded-Proto is: " + forwardedProto); if (forwardedProto != null && !forwardedProto.isEmpty()) { From f32355e61dab77bd8ed2e900c4b4913abb75b5b4 Mon Sep 17 00:00:00 2001 From: Jim Myers Date: Tue, 2 Sep 2025 15:59:02 -0400 Subject: [PATCH 028/578] documentation --- doc/sphinx-guides/source/api/external-tools.rst | 4 ++++ doc/sphinx-guides/source/installation/config.rst | 10 ++++++++++ 2 files changed, 14 insertions(+) diff --git a/doc/sphinx-guides/source/api/external-tools.rst b/doc/sphinx-guides/source/api/external-tools.rst index ae0e44b36aa..0e542198d99 100644 --- a/doc/sphinx-guides/source/api/external-tools.rst +++ b/doc/sphinx-guides/source/api/external-tools.rst @@ -171,6 +171,10 @@ The signed URL mechanism is more secure than exposing API tokens and therefore r - For tools invoked via a GET call, Dataverse will include a callback query parameter with a Base64 encoded value. The decoded value is a signed URL that can be called to retrieve a JSON response containing all of the queryParameters and allowedApiCalls specified in the manfiest. - For tools invoked via POST, Dataverse will send a JSON body including the requested queryParameters and allowedApiCalls. Dataverse expects the response to the POST to indicate a redirect which Dataverse will use to open the tool. +.. note:: + + **For Dataverse site administrators:** When Dataverse is behind a proxy, signed URLs may not work correctly due to protocol mismatches (HTTP vs HTTPS). Please refer to the :ref:`signed-urls-forwarded-proto-header` section to ensure signed URLs work properly in proxy environments. + API Token ^^^^^^^^^ diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 67621e5eb8c..60b9b4170cd 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -94,6 +94,16 @@ First of all, confirm that access is denied! If you are in fact able to access t Still feel like activating this option in your configuration? - Have fun and be safe! +.. _signed-urls-forwarded-proto-header: + +Using X-Forwarded-Proto for Signed Urls ++++++++++++++++++++++++++++++++++++++++ + +If you use an Apache or Nginx proxy, or have a firewall such as Anubis, and they are configured to forward traffic to Dataverse over http +(i.e. your proxy receives user calls over https but forwards locally to Dataverse over http), signed urls, used by external tools and +upload apps (such as DVWebloader), are likely to fail unless you configure your proxy to send an X-ForwardedProto HTTP Header. +This allows Dataverse to recognize that the communication from the user was over https and that validation of signed urls should assume +they started with https:// (rather than http:// as received from the proxy). .. _PrivacyConsiderations: From 33ca52787e07663ea58627f4b20d303c02030786 Mon Sep 17 00:00:00 2001 From: Jim Myers Date: Tue, 2 Sep 2025 16:04:56 -0400 Subject: [PATCH 029/578] release note --- doc/release-notes/11787-signed-url-improvement.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 doc/release-notes/11787-signed-url-improvement.md diff --git a/doc/release-notes/11787-signed-url-improvement.md b/doc/release-notes/11787-signed-url-improvement.md new file mode 100644 index 00000000000..6caadc8523a --- /dev/null +++ b/doc/release-notes/11787-signed-url-improvement.md @@ -0,0 +1 @@ +In prior versions of Dataverse, configuring a proxy to forward to Dataverse over an http connection could result in failure of signed Urls (e.g. for external tools). This version of Dataverse supports having a proxy send an X-Forwarded-Proto header set to https to avoid this issue. \ No newline at end of file From acda8c287552a02b00743b03c064bb0ed8d20933 Mon Sep 17 00:00:00 2001 From: Jim Myers Date: Thu, 4 Sep 2025 13:41:13 -0400 Subject: [PATCH 030/578] fixes per review --- doc/sphinx-guides/source/installation/config.rst | 2 +- .../harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 60b9b4170cd..037c0f82420 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -101,7 +101,7 @@ Using X-Forwarded-Proto for Signed Urls If you use an Apache or Nginx proxy, or have a firewall such as Anubis, and they are configured to forward traffic to Dataverse over http (i.e. your proxy receives user calls over https but forwards locally to Dataverse over http), signed urls, used by external tools and -upload apps (such as DVWebloader), are likely to fail unless you configure your proxy to send an X-ForwardedProto HTTP Header. +upload apps (such as DVWebloader), are likely to fail unless you configure your proxy to send an X-Forwarded-Proto HTTP Header. This allows Dataverse to recognize that the communication from the user was over https and that validation of signed urls should assume they started with https:// (rather than http:// as received from the proxy). diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java index d8457304e18..e701876d5ce 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java @@ -82,7 +82,7 @@ private User getAuthenticatedUserFromSignedUrl(ContainerRequestContext container if (forwardedProto != null && !forwardedProto.isEmpty()) { - if ("https".equalsIgnoreCase(forwardedProto) && signedUrl.startsWith("http:")) { + if ("https".equalsIgnoreCase(forwardedProto) && signedUrl.toLowerCase().startsWith("http:")) { signedUrl = "https" + signedUrl.substring(4); } } From b1c4b467461a41e4cd581df648263ea9cdf297ee Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Tue, 9 Sep 2025 12:09:05 -0400 Subject: [PATCH 031/578] New Dataset Move Notification --- .../11670-notification-of-moved-datasets.md | 5 + .../source/admin/user-administration.rst | 1 + .../harvard/iq/dataverse/MailServiceBean.java | 7 ++ .../iq/dataverse/UserNotification.java | 3 +- .../harvard/iq/dataverse/api/Datasets.java | 3 +- .../providers/builtin/DataverseUserPage.java | 1 + .../dashboard/DashboardMoveDatasetPage.java | 3 +- .../command/impl/MoveDatasetCommand.java | 61 +++++++++-- .../settings/SettingsServiceBean.java | 6 ++ .../harvard/iq/dataverse/util/MailUtil.java | 2 + .../json/InAppNotificationsJsonPrinter.java | 11 ++ src/main/java/propertyFiles/Bundle.properties | 4 + src/main/webapp/dataverseuser.xhtml | 14 +++ .../edu/harvard/iq/dataverse/api/MoveIT.java | 102 +++++++++++++++++- 14 files changed, 208 insertions(+), 15 deletions(-) create mode 100644 doc/release-notes/11670-notification-of-moved-datasets.md diff --git a/doc/release-notes/11670-notification-of-moved-datasets.md b/doc/release-notes/11670-notification-of-moved-datasets.md new file mode 100644 index 00000000000..cf1935e2e88 --- /dev/null +++ b/doc/release-notes/11670-notification-of-moved-datasets.md @@ -0,0 +1,5 @@ +## Notifications + +New notification was added for Datasets moving between Dataverses + +See #11670 diff --git a/doc/sphinx-guides/source/admin/user-administration.rst b/doc/sphinx-guides/source/admin/user-administration.rst index 5efd56fbd33..bd5eb1dcce5 100644 --- a/doc/sphinx-guides/source/admin/user-administration.rst +++ b/doc/sphinx-guides/source/admin/user-administration.rst @@ -98,6 +98,7 @@ This enables additional settings for each user in the notifications tab of their * ``CREATEDS`` Your dataset is created * ``CREATEDV`` Dataverse collection is created * ``DATASETCREATED`` Dataset was created by user +* ``DATASETMOVED`` Dataset was moved by user * ``FILESYSTEMIMPORT`` Dataset has been successfully uploaded and verified * ``GRANTFILEACCESS`` Access to file is granted * ``INGESTCOMPLETEDWITHERRORS`` Ingest completed with errors diff --git a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java index 569bb82ff80..e718ad409a9 100644 --- a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java @@ -478,6 +478,12 @@ public String getMessageTextBasedOnNotification(UserNotification userNotificatio String[] paramArrayDatasetCreated = {getDatasetLink(dataset), dataset.getDisplayName(), userNotification.getRequestor().getName(), dataset.getOwner().getDisplayName()}; messageText += MessageFormat.format(pattern, paramArrayDatasetCreated); return messageText; + case DATASETMOVED: + dataset = (Dataset) targetObject; + pattern = BundleUtil.getStringFromBundle("notification.email.datasetWasMoved"); + String[] paramArrayDatasetMoved = {getDatasetLink(dataset), dataset.getDisplayName(), userNotification.getRequestor().getName(), dataset.getOwner().getDisplayName()}; + messageText += MessageFormat.format(pattern, paramArrayDatasetMoved); + return messageText; case CREATEDS: version = (DatasetVersion) targetObject; String datasetCreatedMessage = BundleUtil.getStringFromBundle("notification.email.createDataset", Arrays.asList( @@ -785,6 +791,7 @@ public Object getObjectOfNotification (UserNotification userNotification){ case GRANTFILEACCESS: case REJECTFILEACCESS: case DATASETCREATED: + case DATASETMOVED: case DATASETMENTIONED: return datasetService.find(userNotification.getObjectId()); case CREATEDS: diff --git a/src/main/java/edu/harvard/iq/dataverse/UserNotification.java b/src/main/java/edu/harvard/iq/dataverse/UserNotification.java index c1cfe72e3cd..67a6e368cb0 100644 --- a/src/main/java/edu/harvard/iq/dataverse/UserNotification.java +++ b/src/main/java/edu/harvard/iq/dataverse/UserNotification.java @@ -40,7 +40,8 @@ public enum Type { PUBLISHFAILED_PIDREG, WORKFLOW_SUCCESS, WORKFLOW_FAILURE, STATUSUPDATED, DATASETCREATED, DATASETMENTIONED, GLOBUSUPLOADCOMPLETED, GLOBUSUPLOADCOMPLETEDWITHERRORS, GLOBUSDOWNLOADCOMPLETED, GLOBUSDOWNLOADCOMPLETEDWITHERRORS, REQUESTEDFILEACCESS, - GLOBUSUPLOADREMOTEFAILURE, GLOBUSUPLOADLOCALFAILURE, PIDRECONCILED; + GLOBUSUPLOADREMOTEFAILURE, GLOBUSUPLOADLOCALFAILURE, PIDRECONCILED, + DATASETMOVED; public String getDescription() { return BundleUtil.getStringFromBundle("notification.typeDescription." + this.name()); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index 729174dedfc..9d3c96d609e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -1396,8 +1396,7 @@ public Response moveDataset(@Context ContainerRequestContext crc, @PathParam("id } //Command requires Super user - it will be tested by the command execCommand(new MoveDatasetCommand( - createDataverseRequest(u), ds, target, force - )); + createDataverseRequest(u), ds, target, force, true)); return ok(BundleUtil.getStringFromBundle("datasets.api.moveDataset.success")); } catch (WrappedResponse ex) { if (ex.getCause() instanceof UnforcedCommandException) { diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java index cda94d25060..94af281e7a4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java @@ -509,6 +509,7 @@ public void displayNotification() { case GRANTFILEACCESS: case REJECTFILEACCESS: case DATASETCREATED: + case DATASETMOVED: case DATASETMENTIONED: userNotification.setTheObject(datasetService.find(userNotification.getObjectId())); break; diff --git a/src/main/java/edu/harvard/iq/dataverse/dashboard/DashboardMoveDatasetPage.java b/src/main/java/edu/harvard/iq/dataverse/dashboard/DashboardMoveDatasetPage.java index b1333b02a46..a4023f41e24 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dashboard/DashboardMoveDatasetPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/dashboard/DashboardMoveDatasetPage.java @@ -153,8 +153,7 @@ public void move(){ HttpServletRequest httpServletRequest = (HttpServletRequest) FacesContext.getCurrentInstance().getExternalContext().getRequest(); DataverseRequest dataverseRequest = new DataverseRequest(authUser, httpServletRequest); commandEngine.submit(new MoveDatasetCommand( - dataverseRequest, ds, target, false - )); + dataverseRequest, ds, target, false, true)); logger.info("Moved " + dsPersistentId + " from " + srcAlias + " to " + dstAlias); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java index 1c3a62ec6de..e2f85a05ee4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java @@ -5,13 +5,10 @@ */ package edu.harvard.iq.dataverse.engine.command.impl; -import edu.harvard.iq.dataverse.Dataset; -import edu.harvard.iq.dataverse.DatasetLinkingDataverse; -import edu.harvard.iq.dataverse.DatasetLock; -import edu.harvard.iq.dataverse.Dataverse; -import edu.harvard.iq.dataverse.Guestbook; +import edu.harvard.iq.dataverse.*; import edu.harvard.iq.dataverse.authorization.Permission; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.engine.command.AbstractVoidCommand; import edu.harvard.iq.dataverse.engine.command.CommandContext; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; @@ -21,11 +18,12 @@ import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; import edu.harvard.iq.dataverse.engine.command.exception.PermissionException; import edu.harvard.iq.dataverse.engine.command.exception.UnforcedCommandException; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.BundleUtil; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; + +import java.sql.Timestamp; +import java.time.Instant; +import java.util.*; import java.util.logging.Logger; /** @@ -44,8 +42,12 @@ public class MoveDatasetCommand extends AbstractVoidCommand { final Dataset moved; final Dataverse destination; final Boolean force; + private boolean allowSelfNotification = false; public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force) { + this( aRequest, moved, destination, force, Boolean.FALSE); + } + public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force, Boolean allowSelfNotification) { super( aRequest, dv("moved", moved), @@ -54,6 +56,7 @@ public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse de this.moved = moved; this.destination = destination; this.force= force; + this.allowSelfNotification = allowSelfNotification; } @Override @@ -147,10 +150,50 @@ public void executeImpl(CommandContext ctxt) throws CommandException { // OK, move moved.setOwner(destination); ctxt.em().merge(moved); + sendNotification(moved, ctxt); boolean doNormalSolrDocCleanUp = true; ctxt.index().asyncIndexDataset(moved, doNormalSolrDocCleanUp); } + /** + * Sends notifications to those able to publish the dataset upon the successful move of a dataset. + *

+ * This method checks if dataset move notifications are enabled. If so, it + * notifies all users with {@code Permission.PublishDataset} on the dataset. + * The user who initiated the action can be included or excluded from this + * notification based on the allowSelfNotification flag. + * + * @param dataset The moved {@code Dataset}. + * @param ctxt The {@code CommandContext} providing access to application services. + */ + protected void sendNotification(Dataset dataset, CommandContext ctxt) { + // 1. Exit early if the SendNotificationOnDatasetMove setting is disabled. + if (!ctxt.settings().isTrueForKey(SettingsServiceBean.Key.SendNotificationOnDatasetMove, false)) { + return; + } + + // 2. Identify the user who initiated the action. + final User user = getUser(); + final AuthenticatedUser requestor = user.isAuthenticated() ? (AuthenticatedUser) user : null; + // 3. Get all users with publish permission and notify them. + Map recipients = ctxt.permissions().getDistinctUsersWithPermissionOn(Permission.PublishDataset, dataset); + if (requestor != null) { + recipients.put(requestor.getIdentifier(), requestor); + } + + recipients.values() + .stream() + .filter(recipient -> allowSelfNotification || !recipient.equals(requestor)) + .forEach(recipient -> ctxt.notifications().sendNotification( + recipient, + Timestamp.from(Instant.now()), + UserNotification.Type.DATASETMOVED, + dataset.getId(), + null, + requestor, + true + )); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java index b323a9b7861..c48c280abce 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java @@ -573,6 +573,12 @@ Whether Harvesting (OAI) service is enabled * ability/permission necessary to publish the dataset */ SendNotificationOnDatasetCreation, + /** + * A boolean setting that, if true will send an email and notification to users + * when a Dataset is moved. Messages go to those who have the + * ability/permission necessary to publish the dataset + */ + SendNotificationOnDatasetMove, /** * A JSON Object containing named comma separated sets(s) of allowed labels (up * to 32 characters, spaces allowed) that can be set on draft datasets, via API diff --git a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java index fe309dbe345..5482c4bdbac 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java @@ -47,6 +47,8 @@ public static String getSubjectTextBasedOnNotification(UserNotification userNoti return BundleUtil.getStringFromBundle("notification.email.rejected.file.access.subject", rootDvNameAsList); case DATASETCREATED: return BundleUtil.getStringFromBundle("notification.email.dataset.created.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); + case DATASETMOVED: + return BundleUtil.getStringFromBundle("notification.email.dataset.moved.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case CREATEDS: return BundleUtil.getStringFromBundle("notification.email.create.dataset.subject", Arrays.asList(rootDvNameAsList.get(0), datasetDisplayName)); case SUBMITTEDDS: diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java index f3c3a0383e0..306cede0d32 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java @@ -7,6 +7,8 @@ import jakarta.ejb.EJB; import jakarta.ejb.Stateless; +import java.util.logging.Logger; + import static edu.harvard.iq.dataverse.dataset.DatasetUtil.getLocaleCurationStatusLabel; import static edu.harvard.iq.dataverse.util.json.JsonPrinter.jsonRoleAssignments; @@ -19,6 +21,7 @@ @Stateless public class InAppNotificationsJsonPrinter { + private static final Logger logger = Logger.getLogger(InAppNotificationsJsonPrinter.class.getCanonicalName()); public static final String KEY_ROLE_ASSIGNMENTS = "roleAssignments"; public static final String KEY_DATAVERSE_ALIAS = "dataverseAlias"; public static final String KEY_DATAVERSE_DISPLAY_NAME = "dataverseDisplayName"; @@ -125,6 +128,9 @@ public void addFieldsByType(final NullSafeJsonBuilder notificationJson, final Au case DATASETMENTIONED: addDatasetMentionedFields(notificationJson, userNotification); break; + case DATASETMOVED: + addDatasetMovedFields(notificationJson, userNotification, requestor); + break; } } @@ -262,4 +268,9 @@ private void addDatasetMentionedFields(final NullSafeJsonBuilder notificationJso addDatasetFields(notificationJson, userNotification); notificationJson.add(KEY_ADDITIONAL_INFO, userNotification.getAdditionalInfo()); } + + private void addDatasetMovedFields(final NullSafeJsonBuilder notificationJson, final UserNotification userNotification, final AuthenticatedUser requestor) { + addDatasetFields(notificationJson, userNotification); + addRequestorFields(notificationJson, requestor); + } } diff --git a/src/main/java/propertyFiles/Bundle.properties b/src/main/java/propertyFiles/Bundle.properties index d527ba3eeeb..42db152b4b1 100644 --- a/src/main/java/propertyFiles/Bundle.properties +++ b/src/main/java/propertyFiles/Bundle.properties @@ -242,6 +242,7 @@ notification.createDataverse={0} was created in {1} . To learn more about what y notification.dataverse.management.title=Dataverse Management - Dataverse User Guide notification.createDataset={0} was created in {1}. To learn more about what you can do with a dataset, check out the {2}. notification.datasetCreated={0} was created in {1} by {2}. +notification.datasetMoved={0} was moved to {1} by {2}. notification.dataset.management.title=Dataset Management - Dataset User Guide notification.wasSubmittedForReview={0} was submitted for review to be published in {1}. Don''t forget to publish it or send it back to the contributor, {2} ({3})\! notification.wasReturnedByReviewer={0} was returned by the curator of {1}. @@ -315,6 +316,7 @@ notification.typeDescription.WORKFLOW_FAILURE=External workflow run has failed notification.typeDescription.STATUSUPDATED=Status of dataset has been updated notification.typeDescription.DATASETCREATED=Dataset was created by user notification.typeDescription.DATASETMENTIONED=Dataset was referenced in remote system +notification.typeDescription.DATASETMOVED=Dataset was moved by user notification.typeDescription.PIDRECONCILED=The Persistent identifier of dataset has been updated notification.typeDescription.GLOBUSUPLOADCOMPLETED=Globus upload is completed notification.typeDescription.GLOBUSUPLOADCOMPLETEDWITHERRORS=Globus upload completed with errors @@ -815,6 +817,7 @@ dashboard.move.dataverse.menu.invalidMsg=No matches found notification.email.create.dataverse.subject={0}: Your dataverse has been created notification.email.create.dataset.subject={0}: Dataset "{1}" has been created notification.email.dataset.created.subject={0}: Dataset "{1}" has been created +notification.email.dataset.moved.subject={0}: Dataset "{1}" has been moved notification.email.request.file.access.subject={0}: {1} {2} ({3}) requested access to dataset "{4}" notification.email.requested.file.access.subject={0}: You have requested access to a restricted file in dataset "{1}" notification.email.grant.file.access.subject={0}: You have been granted access to a restricted file @@ -865,6 +868,7 @@ notification.email.changeEmail=Hello, {0}.{1}\n\nPlease contact us if you did no notification.email.passwordReset=Hi {0},\n\nSomeone, hopefully you, requested a password reset for {1}.\n\nPlease click the link below to reset your Dataverse account password:\n\n {2} \n\n The link above will only work for the next {3} minutes.\n\n Please contact us if you did not request this password reset or need further help. notification.email.passwordReset.subject=Dataverse Password Reset Requested notification.email.datasetWasCreated=Dataset "{1}" was just created by {2} in the {3} collection. +notification.email.datasetWasMoved=Dataset "{1}" was just moved by {2} to the {3} collection. notification.email.requestedFileAccess=You have requested access to a file(s) in dataset "{1}". Your request has been sent to the managers of this dataset who will grant or reject your request. If you have any questions, you may reach the dataset managers using the "Contact" link on the upper right corner of the dataset page. hours=hours hour=hour diff --git a/src/main/webapp/dataverseuser.xhtml b/src/main/webapp/dataverseuser.xhtml index 6c7826783d7..8be87a89a6b 100644 --- a/src/main/webapp/dataverseuser.xhtml +++ b/src/main/webapp/dataverseuser.xhtml @@ -166,6 +166,20 @@ + + + + + #{item.theObject.getDisplayName()} + + + #{item.theObject.getOwner().getDisplayName()} + + + #{item.requestor.name} + + + diff --git a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java index 8951b0bd42e..edcbd268d75 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java @@ -1,5 +1,7 @@ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import edu.harvard.iq.dataverse.util.BundleUtil; import io.restassured.RestAssured; import io.restassured.path.json.JsonPath; import static io.restassured.path.json.JsonPath.with; @@ -7,6 +9,7 @@ import edu.harvard.iq.dataverse.authorization.DataverseRole; import java.io.StringReader; import java.util.List; +import java.util.Map; import java.util.logging.Logger; import jakarta.json.Json; import jakarta.json.JsonObject; @@ -15,9 +18,13 @@ import static jakarta.ws.rs.core.Response.Status.FORBIDDEN; import static jakarta.ws.rs.core.Response.Status.OK; import static jakarta.ws.rs.core.Response.Status.UNAUTHORIZED; + import org.hamcrest.CoreMatchers; -import static org.hamcrest.CoreMatchers.equalTo; + +import static org.hamcrest.CoreMatchers.*; import static org.junit.jupiter.api.Assertions.assertEquals; + +import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; @@ -29,6 +36,10 @@ public class MoveIT { public static void setUpClass() { RestAssured.baseURI = UtilIT.getRestAssuredBaseUri(); } + @AfterAll + public static void afterClass() { + sendNotificationOnDatasetMoveSetting(false); + } @Test public void testMoveDataset() { @@ -145,6 +156,84 @@ public void testMoveDataset() { } + @Test + public void testMoveDatasetNotification() { + sendNotificationOnDatasetMoveSetting(true); + // Create the first user/dataverse/dataset + Response createUser1 = UtilIT.createRandomUser(); + createUser1.prettyPrint(); + createUser1.then().assertThat() + .statusCode(OK.getStatusCode()); + String user1Username = UtilIT.getUsernameFromResponse(createUser1); + String user1ApiToken = UtilIT.getApiTokenFromResponse(createUser1); + UtilIT.setSuperuserStatus(user1Username, true); + + Response createDataverse1 = UtilIT.createRandomDataverse(user1ApiToken); + createDataverse1.prettyPrint(); + createDataverse1.then().assertThat() + .statusCode(CREATED.getStatusCode()); + String dataverseAlias1 = UtilIT.getAliasFromResponse(createDataverse1); + + Response createDataset = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias1, user1ApiToken); + createDataset.prettyPrint(); + createDataset.then().assertThat() + .statusCode(CREATED.getStatusCode()); + Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + + // Create the second user/dataverse + Response createUser2 = UtilIT.createRandomUser(); + createUser2.prettyPrint(); + createUser2.then().assertThat() + .statusCode(OK.getStatusCode()); + String user2Username = UtilIT.getUsernameFromResponse(createUser2); + String user2ApiToken = UtilIT.getApiTokenFromResponse(createUser2); + + Response createDataverse2 = UtilIT.createRandomDataverse(user2ApiToken); + createDataverse2.prettyPrint(); + createDataverse2.then().assertThat() + .statusCode(CREATED.getStatusCode()); + String dataverseAlias2 = UtilIT.getAliasFromResponse(createDataverse2); + + // clear existing notifications + clearNotifications(user1ApiToken); + clearNotifications(user2ApiToken); + + // Move the dataset from dataverse1 to dataverse2 + Response moveDataset = UtilIT.moveDataset(datasetId.toString(), dataverseAlias2, user1ApiToken); + moveDataset.prettyPrint(); + moveDataset.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.message", equalTo(BundleUtil.getStringFromBundle("datasets.api.moveDataset.success"))); + + // verify that a notification was sent to user1 + Response getNotifications = UtilIT.getNotifications(user1ApiToken); + getNotifications.prettyPrint(); + getNotifications.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.notifications[0].type", equalTo("DATASETMOVED")) + .body("data.notifications[0].displayAsRead", equalTo(false)) + .body("data.notifications[0].subjectText", containsString("has been moved")) + .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + // verify that a notification was sent to user2 + getNotifications = UtilIT.getNotifications(user2ApiToken); + getNotifications.prettyPrint(); + getNotifications.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.notifications[0].type", equalTo("DATASETMOVED")) + .body("data.notifications[0].displayAsRead", equalTo(false)) + .body("data.notifications[0].subjectText", containsString("has been moved")) + .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + } + + private void clearNotifications(String apiToken) { + Response getNotifications = UtilIT.getNotifications(apiToken); + List notifications = JsonPath.from(getNotifications.body().asString()).getList("data.notifications"); + for (Object obj : notifications) { + Object id = ((Map) obj).get("id"); + UtilIT.deleteNotification(Long.parseLong(id.toString()), apiToken).prettyPrint(); + } + } + @Test public void testMoveDatasetThief() { @@ -408,4 +497,15 @@ public void testMoveDatasetsPerms() { } + private static void sendNotificationOnDatasetMoveSetting(boolean enable) { + Response resp; + if (enable) { + resp = UtilIT.enableSetting(SettingsServiceBean.Key.SendNotificationOnDatasetMove); + } else { + resp = UtilIT.deleteSetting(SettingsServiceBean.Key.SendNotificationOnDatasetMove); + } + resp.prettyPrint(); + resp.then().assertThat() + .statusCode(OK.getStatusCode()); + } } From 1766c5e8e634fe26df07cc4d6521d48f0c9cd43e Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Tue, 9 Sep 2025 15:15:50 -0400 Subject: [PATCH 032/578] remove unused logger --- .../iq/dataverse/util/json/InAppNotificationsJsonPrinter.java | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java index 306cede0d32..902f820109c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/InAppNotificationsJsonPrinter.java @@ -7,8 +7,6 @@ import jakarta.ejb.EJB; import jakarta.ejb.Stateless; -import java.util.logging.Logger; - import static edu.harvard.iq.dataverse.dataset.DatasetUtil.getLocaleCurationStatusLabel; import static edu.harvard.iq.dataverse.util.json.JsonPrinter.jsonRoleAssignments; @@ -21,7 +19,6 @@ @Stateless public class InAppNotificationsJsonPrinter { - private static final Logger logger = Logger.getLogger(InAppNotificationsJsonPrinter.class.getCanonicalName()); public static final String KEY_ROLE_ASSIGNMENTS = "roleAssignments"; public static final String KEY_DATAVERSE_ALIAS = "dataverseAlias"; public static final String KEY_DATAVERSE_DISPLAY_NAME = "dataverseDisplayName"; From e785c0bb78079563c6c5c3bcb5e7954a7af49243 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Tue, 9 Sep 2025 15:34:46 -0400 Subject: [PATCH 033/578] fix docs --- doc/release-notes/11670-notification-of-moved-datasets.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/release-notes/11670-notification-of-moved-datasets.md b/doc/release-notes/11670-notification-of-moved-datasets.md index cf1935e2e88..bbec87a79fd 100644 --- a/doc/release-notes/11670-notification-of-moved-datasets.md +++ b/doc/release-notes/11670-notification-of-moved-datasets.md @@ -1,5 +1,6 @@ ## Notifications -New notification was added for Datasets moving between Dataverses +New notification was added for Datasets moving between Dataverses. +Requires SettingsServiceBean.Key.SendNotificationOnDatasetMove setting to be enabled. See #11670 From 45b7571de450102106679f984c7cfd2a98829867 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Tue, 9 Sep 2025 17:39:57 -0400 Subject: [PATCH 034/578] fix notification user dataset original owner instead of new owner to check permissions --- .../command/impl/MoveDatasetCommand.java | 13 +++++++----- .../edu/harvard/iq/dataverse/api/MoveIT.java | 21 ++++++++++--------- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java index e2f85a05ee4..513e936dabf 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java @@ -148,9 +148,10 @@ public void executeImpl(CommandContext ctxt) throws CommandException { } // OK, move + Dataverse originalOwner = moved.getOwner(); moved.setOwner(destination); ctxt.em().merge(moved); - sendNotification(moved, ctxt); + sendNotification(moved, originalOwner, ctxt); boolean doNormalSolrDocCleanUp = true; ctxt.index().asyncIndexDataset(moved, doNormalSolrDocCleanUp); @@ -160,14 +161,15 @@ public void executeImpl(CommandContext ctxt) throws CommandException { * Sends notifications to those able to publish the dataset upon the successful move of a dataset. *

* This method checks if dataset move notifications are enabled. If so, it - * notifies all users with {@code Permission.PublishDataset} on the dataset. + * notifies all users with {@code Permission.PublishDataset} on the original owning Dataverse. * The user who initiated the action can be included or excluded from this * notification based on the allowSelfNotification flag. * * @param dataset The moved {@code Dataset}. + * @param originalOwner The original owning {@code Dataverse}. * @param ctxt The {@code CommandContext} providing access to application services. */ - protected void sendNotification(Dataset dataset, CommandContext ctxt) { + protected void sendNotification(Dataset dataset, Dataverse originalOwner, CommandContext ctxt) { // 1. Exit early if the SendNotificationOnDatasetMove setting is disabled. if (!ctxt.settings().isTrueForKey(SettingsServiceBean.Key.SendNotificationOnDatasetMove, false)) { return; @@ -177,8 +179,9 @@ protected void sendNotification(Dataset dataset, CommandContext ctxt) { final User user = getUser(); final AuthenticatedUser requestor = user.isAuthenticated() ? (AuthenticatedUser) user : null; - // 3. Get all users with publish permission and notify them. - Map recipients = ctxt.permissions().getDistinctUsersWithPermissionOn(Permission.PublishDataset, dataset); + // 3. Get all users with publish permission on the dataset's original owner (dataverse) and notify them. + Map recipients = ctxt.permissions().getDistinctUsersWithPermissionOn(Permission.PublishDataset, originalOwner); + // make sure the requestor is in the recipient list in case they don't match the permission if (requestor != null) { recipients.put(requestor.getIdentifier(), requestor); } diff --git a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java index edcbd268d75..a547c44eea2 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java @@ -159,7 +159,7 @@ public void testMoveDataset() { @Test public void testMoveDatasetNotification() { sendNotificationOnDatasetMoveSetting(true); - // Create the first user/dataverse/dataset + // Create the first user/dataverse (superuser) Response createUser1 = UtilIT.createRandomUser(); createUser1.prettyPrint(); createUser1.then().assertThat() @@ -174,12 +174,6 @@ public void testMoveDatasetNotification() { .statusCode(CREATED.getStatusCode()); String dataverseAlias1 = UtilIT.getAliasFromResponse(createDataverse1); - Response createDataset = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias1, user1ApiToken); - createDataset.prettyPrint(); - createDataset.then().assertThat() - .statusCode(CREATED.getStatusCode()); - Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); - // Create the second user/dataverse Response createUser2 = UtilIT.createRandomUser(); createUser2.prettyPrint(); @@ -194,12 +188,19 @@ public void testMoveDatasetNotification() { .statusCode(CREATED.getStatusCode()); String dataverseAlias2 = UtilIT.getAliasFromResponse(createDataverse2); - // clear existing notifications + // User2 creates dataset in DV2 + Response createDataset = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias2, user2ApiToken); + createDataset.prettyPrint(); + createDataset.then().assertThat() + .statusCode(CREATED.getStatusCode()); + Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + + // clear existing notifications (so the DATASETMOVED notification will be the only one) clearNotifications(user1ApiToken); clearNotifications(user2ApiToken); - // Move the dataset from dataverse1 to dataverse2 - Response moveDataset = UtilIT.moveDataset(datasetId.toString(), dataverseAlias2, user1ApiToken); + // User1(superuser) moves the dataset from dataverse2 to dataverse1 + Response moveDataset = UtilIT.moveDataset(datasetId.toString(), dataverseAlias1, user1ApiToken); moveDataset.prettyPrint(); moveDataset.then().assertThat() .statusCode(OK.getStatusCode()) From d3a1d58e4c3e41f42788c454b9e92a97b51c6d14 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Mon, 15 Sep 2025 15:30:30 -0400 Subject: [PATCH 035/578] allow api call to ignore setting --- .../harvard/iq/dataverse/api/Datasets.java | 23 +++++--- .../iq/dataverse/util/json/JsonPrinter.java | 30 ++++++----- .../harvard/iq/dataverse/api/DatasetsIT.java | 53 +++++++++++++++---- .../edu/harvard/iq/dataverse/api/UtilIT.java | 13 ++++- 4 files changed, 88 insertions(+), 31 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index fb97c761ef1..9ea7692de78 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -484,12 +484,16 @@ public Response getVersion(@Context ContainerRequestContext crc, @QueryParam("excludeMetadataBlocks") Boolean excludeMetadataBlocks, @QueryParam("includeDeaccessioned") boolean includeDeaccessioned, @QueryParam("returnOwners") boolean returnOwners, + @QueryParam("ignoreSettingExcludeEmailFromExport") Boolean ignoreSettingToExcludeEmailFromExport, @Context UriInfo uriInfo, @Context HttpHeaders headers) { return response( req -> { + boolean includeMetadataBlocks = excludeMetadataBlocks == null ? true : !excludeMetadataBlocks; + boolean includeFiles = excludeFiles == null ? true : !excludeFiles; + boolean ignoreSettingExcludeEmailFromExport = ignoreSettingToExcludeEmailFromExport != null ? ignoreSettingToExcludeEmailFromExport : false; //If excludeFiles is null the default is to provide the files and because of this we need to check permissions. - boolean checkPerms = excludeFiles == null ? true : !excludeFiles; + boolean checkPerms = includeFiles; Dataset dataset = findDatasetOrDie(datasetId); DatasetVersion requestedDatasetVersion = getDatasetVersionOrDie(req, @@ -503,16 +507,19 @@ public Response getVersion(@Context ContainerRequestContext crc, if (requestedDatasetVersion == null || requestedDatasetVersion.getId() == null) { return notFound("Dataset version not found"); } - - if (excludeFiles == null ? true : !excludeFiles) { + if (includeFiles) { requestedDatasetVersion = datasetversionService.findDeep(requestedDatasetVersion.getId()); } - Boolean includeMetadataBlocks = excludeMetadataBlocks == null ? true : !excludeMetadataBlocks; - JsonObjectBuilder jsonBuilder = json(requestedDatasetVersion, - null, - excludeFiles == null ? true : !excludeFiles, - returnOwners, includeMetadataBlocks); + // Check to see if the caller wants to ignore the ExcludeEmailFromExport setting in the metadata block and that they have permission to do so + // Let the JsonPrinter know to ignore the ExcludeEmailFromExport setting so the emails will show for this API call by permitted user + if (ignoreSettingExcludeEmailFromExport && (!includeMetadataBlocks || !permissionService.userOn(getRequestUser(crc), dataset).has(Permission.EditDataset))) { + // either not showing metadata block or user isn't allowed to override the setting + ignoreSettingExcludeEmailFromExport = false; + } + + JsonObjectBuilder jsonBuilder = json(requestedDatasetVersion, null, includeFiles, + returnOwners, includeMetadataBlocks, ignoreSettingExcludeEmailFromExport); return ok(jsonBuilder); }, getRequestUser(crc)); diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index bbc834e0cc4..2f57b8fb2bd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -495,17 +495,17 @@ public static JsonObjectBuilder json(FileDetailsHolder ds) { } public static JsonObjectBuilder json(DatasetVersion dsv, boolean includeFiles) { - return json(dsv, null, includeFiles, false,true); + return json(dsv, null, includeFiles, false, true, false); } public static JsonObjectBuilder json(DatasetVersion dsv, boolean includeFiles, boolean includeMetadataBlocks) { - return json(dsv, null, includeFiles, false, includeMetadataBlocks); + return json(dsv, null, includeFiles, false, includeMetadataBlocks, false); } public static JsonObjectBuilder json(DatasetVersion dsv, List anonymizedFieldTypeNamesList, boolean includeFiles, boolean returnOwners) { - return json( dsv, anonymizedFieldTypeNamesList, includeFiles, returnOwners,true); + return json(dsv, anonymizedFieldTypeNamesList, includeFiles, returnOwners, true, false); } public static JsonObjectBuilder json(DatasetVersion dsv, List anonymizedFieldTypeNamesList, - boolean includeFiles, boolean returnOwners, boolean includeMetadataBlocks) { + boolean includeFiles, boolean returnOwners, boolean includeMetadataBlocks, boolean ignoreSettingExcludeEmailFromExport) { Dataset dataset = dsv.getDataset(); JsonObjectBuilder bld = jsonObjectBuilder() .add("id", dsv.getId()).add("datasetId", dataset.getId()) @@ -554,10 +554,8 @@ public static JsonObjectBuilder json(DatasetVersion dsv, List anonymized .add("studyCompletion", dsv.getTermsOfUseAndAccess().getStudyCompletion()) .add("fileAccessRequest", dsv.getTermsOfUseAndAccess().isFileAccessRequest()); if(includeMetadataBlocks) { - bld.add("metadataBlocks", (anonymizedFieldTypeNamesList != null) ? - jsonByBlocks(dsv.getDatasetFields(), anonymizedFieldTypeNamesList) - : jsonByBlocks(dsv.getDatasetFields()) - ); + bld.add("metadataBlocks", + jsonByBlocks(dsv.getDatasetFields(), anonymizedFieldTypeNamesList, ignoreSettingExcludeEmailFromExport)); } if(returnOwners){ bld.add("isPartOf", getOwnersFromDvObject(dataset)); @@ -641,15 +639,15 @@ public static JsonObjectBuilder json(DatasetDistributor dist) { } public static JsonObjectBuilder jsonByBlocks(List fields) { - return jsonByBlocks(fields, null); + return jsonByBlocks(fields, null, false); } - public static JsonObjectBuilder jsonByBlocks(List fields, List anonymizedFieldTypeNamesList) { + public static JsonObjectBuilder jsonByBlocks(List fields, List anonymizedFieldTypeNamesList, boolean ignoreSettingExcludeEmailFromExport) { JsonObjectBuilder blocksBld = jsonObjectBuilder(); for (Map.Entry> blockAndFields : DatasetField.groupByBlock(fields).entrySet()) { MetadataBlock block = blockAndFields.getKey(); - blocksBld.add(block.getName(), JsonPrinter.json(block, blockAndFields.getValue(), anonymizedFieldTypeNamesList)); + blocksBld.add(block.getName(), JsonPrinter.json(block, blockAndFields.getValue(), anonymizedFieldTypeNamesList, ignoreSettingExcludeEmailFromExport)); } return blocksBld; } @@ -667,6 +665,10 @@ public static JsonObjectBuilder json(MetadataBlock block, List fie } public static JsonObjectBuilder json(MetadataBlock block, List fields, List anonymizedFieldTypeNamesList) { + return json(block, fields, anonymizedFieldTypeNamesList, false); + } + + public static JsonObjectBuilder json(MetadataBlock block, List fields, List anonymizedFieldTypeNamesList, boolean ignoreSettingExcludeEmailFromExport) { JsonObjectBuilder blockBld = jsonObjectBuilder(); blockBld.add("displayName", block.getDisplayName()); @@ -674,7 +676,11 @@ public static JsonObjectBuilder json(MetadataBlock block, List fie final JsonArrayBuilder fieldsArray = Json.createArrayBuilder(); Map cvocMap = (datasetFieldService==null) ? new HashMap() :datasetFieldService.getCVocConf(true); - DatasetFieldWalker.walk(fields, settingsService, cvocMap, new DatasetFieldsToJson(fieldsArray, anonymizedFieldTypeNamesList)); + if (ignoreSettingExcludeEmailFromExport) { + DatasetFieldWalker.walk(fields, null, cvocMap, new DatasetFieldsToJson(fieldsArray, anonymizedFieldTypeNamesList)); + } else { + DatasetFieldWalker.walk(fields, settingsService, cvocMap, new DatasetFieldsToJson(fieldsArray, anonymizedFieldTypeNamesList)); + } blockBld.add("fields", fieldsArray); return blockBld; diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java index b273502e6ed..a8d1b9bb48b 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java @@ -31,10 +31,7 @@ import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.exception.ExceptionUtils; import org.hamcrest.CoreMatchers; -import org.junit.jupiter.api.AfterAll; -import org.junit.jupiter.api.BeforeAll; -import org.junit.jupiter.api.Disabled; -import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.*; import org.skyscreamer.jsonassert.JSONAssert; import javax.xml.stream.XMLInputFactory; @@ -101,6 +98,10 @@ public static void setUpClass() { */ } + @AfterEach + public void afterEach() { + UtilIT.deleteSetting(SettingsServiceBean.Key.ExcludeEmailFromExport); + } @AfterAll public static void afterClass() { @@ -1757,11 +1758,6 @@ public void testExcludeEmail() { Response deleteUserResponse = UtilIT.deleteUser(username); deleteUserResponse.prettyPrint(); assertEquals(200, deleteUserResponse.getStatusCode()); - - Response removeExcludeEmail = UtilIT.deleteSetting(SettingsServiceBean.Key.ExcludeEmailFromExport); - removeExcludeEmail.then().assertThat() - .statusCode(200); - } @Disabled @@ -6905,6 +6901,45 @@ public void testUpdateMultipleFileMetadata() { .statusCode(OK.getStatusCode()); } + @Test + public void testExcludeEmailOverride() { + // Create super user + String apiToken = getSuperuserToken(); + // Create user with no permission + String apiTokenNoPerms = UtilIT.createRandomUserGetToken(); + // Create Collection + String collectionAlias = UtilIT.createRandomCollectionGetAlias(apiToken); + // Publish Collection + UtilIT.publishDataverseViaNativeApi(collectionAlias, apiToken).prettyPrint(); + // Create Dataset + Response createDataset = UtilIT.createRandomDatasetViaNativeApi(collectionAlias, apiToken); + createDataset.then().assertThat() + .statusCode(CREATED.getStatusCode()); + Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + String datasetPid = JsonPath.from(createDataset.asString()).getString("data.persistentId"); + // Publish Dataset + UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken).prettyPrint(); + + UtilIT.setSetting(SettingsServiceBean.Key.ExcludeEmailFromExport, "true"); + // User has permission to ignore the setting allowing the datasetContactEmail to be included in the response + Response response = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_LATEST_PUBLISHED, apiToken, true, false, false, true); + //response.prettyPrint(); + response.then().assertThat() + .statusCode(OK.getStatusCode()); + + String json = response.prettyPrint(); + assertTrue(json.contains("datasetContactEmail")); + + // User has no permission to override the setting. datasetContactEmail will be excluded + response = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_LATEST_PUBLISHED, apiTokenNoPerms, true, false, false, true); + //response.prettyPrint(); + response.then().assertThat() + .statusCode(OK.getStatusCode()); + + json = response.prettyPrint(); + assertTrue(!json.contains("datasetContactEmail")); + } + private String getSuperuserToken() { Response createResponse = UtilIT.createRandomUser(); String adminApiToken = UtilIT.getApiTokenFromResponse(createResponse); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java index eba8181e566..e2176d448d4 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java @@ -1753,7 +1753,15 @@ static Response getDatasetVersion(String persistentId, String versionNumber, Str static Response getDatasetVersion(String persistentId, String versionNumber, String apiToken, boolean excludeFiles, boolean includeDeaccessioned) { return getDatasetVersion(persistentId,versionNumber,apiToken,excludeFiles,false,includeDeaccessioned); } - static Response getDatasetVersion(String persistentId, String versionNumber, String apiToken, boolean excludeFiles,boolean excludeMetadataBlocks, boolean includeDeaccessioned) { + static Response getDatasetVersion(String persistentId, String versionNumber, String apiToken, boolean excludeFiles, boolean excludeMetadataBlocks, boolean includeDeaccessioned) { + return getDatasetVersion(persistentId, versionNumber, apiToken, excludeFiles, excludeMetadataBlocks, includeDeaccessioned, false); + } + // includeMetadataBlocksEmail is an override of the Setting ExcludeEmailFromExport. excludeMetadataBlocks must be false and user needs EditDataset permission + static Response getDatasetVersion(String persistentId, String versionNumber, String apiToken, + boolean excludeFiles, + boolean excludeMetadataBlocks, + boolean includeDeaccessioned, + boolean ignoreSettingExcludeEmailFromExport) { return given() .header(API_TOKEN_HTTP_HEADER, apiToken) .queryParam("includeDeaccessioned", includeDeaccessioned) @@ -1762,7 +1770,8 @@ static Response getDatasetVersion(String persistentId, String versionNumber, Str + "?persistentId=" + persistentId + (excludeFiles ? "&excludeFiles=true" : "") - + (excludeMetadataBlocks ? "&excludeMetadataBlocks=true" : "")); + + (excludeMetadataBlocks ? "&excludeMetadataBlocks=true" : "") + + (ignoreSettingExcludeEmailFromExport ? "&ignoreSettingExcludeEmailFromExport=true" : "")); } static Response compareDatasetVersions(String persistentId, String versionNumber1, String versionNumber2, String apiToken) { return given() From 58652bfc7b3fa1fa67339b6d11e4e4701efe6f2a Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Mon, 15 Sep 2025 15:51:06 -0400 Subject: [PATCH 036/578] add release note --- ...l-from-export-for-permitted-user-get-dataset-version.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 doc/release-notes/11714-prevent-exclude-email-from-export-for-permitted-user-get-dataset-version.md diff --git a/doc/release-notes/11714-prevent-exclude-email-from-export-for-permitted-user-get-dataset-version.md b/doc/release-notes/11714-prevent-exclude-email-from-export-for-permitted-user-get-dataset-version.md new file mode 100644 index 00000000000..6a87c065573 --- /dev/null +++ b/doc/release-notes/11714-prevent-exclude-email-from-export-for-permitted-user-get-dataset-version.md @@ -0,0 +1,7 @@ +New query parameter (ignoreSettingExcludeEmailFromExport) for API /api/datasets/:persistentId/versions/{versionId} + +SPA requires the ability to have the contact emails included in the response for this API call +This query parameter prevents the contact email from being excluded when the setting (ExcludeEmailFromExport) is set to true and the user has EditDataset permissions. + +See: +- [#11714](https://github.com/IQSS/dataverse/issues/11714) From d3cf158cfada6a6e35e899220fee5072e39fa8cb Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Mon, 15 Sep 2025 16:08:44 -0400 Subject: [PATCH 037/578] adding more tests --- .../harvard/iq/dataverse/api/DatasetsIT.java | 33 ++++++++++++------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java index a8d1b9bb48b..9f647349832 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java @@ -6920,23 +6920,34 @@ public void testExcludeEmailOverride() { // Publish Dataset UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken).prettyPrint(); + // Setting is not set - datasetContactEmail will NOT be excluded + Response response = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_LATEST_PUBLISHED, apiToken); + response.then().assertThat().statusCode(OK.getStatusCode()); + String json = response.prettyPrint(); + assertTrue(json.contains("datasetContactName")); + assertTrue(json.contains("datasetContactEmail")); + UtilIT.setSetting(SettingsServiceBean.Key.ExcludeEmailFromExport, "true"); - // User has permission to ignore the setting allowing the datasetContactEmail to be included in the response - Response response = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_LATEST_PUBLISHED, apiToken, true, false, false, true); - //response.prettyPrint(); - response.then().assertThat() - .statusCode(OK.getStatusCode()); - String json = response.prettyPrint(); + // User does not ignore the setting - datasetContactEmail will be excluded + response = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_LATEST_PUBLISHED, apiToken); + response.then().assertThat().statusCode(OK.getStatusCode()); + json = response.prettyPrint(); + assertTrue(json.contains("datasetContactName")); + assertTrue(!json.contains("datasetContactEmail")); + + // User has permission to ignore the setting allowing the datasetContactEmail to be included in the response + response = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_LATEST_PUBLISHED, apiToken, true, false, false, true); + response.then().assertThat().statusCode(OK.getStatusCode()); + json = response.prettyPrint(); + assertTrue(json.contains("datasetContactName")); assertTrue(json.contains("datasetContactEmail")); - // User has no permission to override the setting. datasetContactEmail will be excluded + // User has no permission to override the setting - datasetContactEmail will be excluded response = UtilIT.getDatasetVersion(datasetPid, DS_VERSION_LATEST_PUBLISHED, apiTokenNoPerms, true, false, false, true); - //response.prettyPrint(); - response.then().assertThat() - .statusCode(OK.getStatusCode()); - + response.then().assertThat().statusCode(OK.getStatusCode()); json = response.prettyPrint(); + assertTrue(json.contains("datasetContactName")); assertTrue(!json.contains("datasetContactEmail")); } From 86bb6d0727614c806a011419396cf17943b705a5 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 25 Sep 2025 09:18:19 -0400 Subject: [PATCH 038/578] clean up use of session for spa --- .../edu/harvard/iq/dataverse/api/Access.java | 139 +++++------------- 1 file changed, 36 insertions(+), 103 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Access.java b/src/main/java/edu/harvard/iq/dataverse/api/Access.java index 89a4cd743d7..10ed320ec76 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Access.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Access.java @@ -15,7 +15,6 @@ import edu.harvard.iq.dataverse.authorization.Permission; import edu.harvard.iq.dataverse.authorization.RoleAssignee; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; -import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser; import edu.harvard.iq.dataverse.authorization.users.GuestUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.dataaccess.DataAccess; @@ -185,14 +184,12 @@ public BundleDownloadInstance datafileBundle(@Context ContainerRequestContext cr DataFile df = findDataFileOrDieWrapper(fileId); - // This will throw a ForbiddenException if access isn't authorized: - checkAuthorization(getRequestUser(crc), df); + // This will throw a ForbiddenException if access isn't authorized: + checkAuthorization(crc, df); if (gbrecs != true && df.isReleased()){ // Write Guestbook record if not done previously and file is released - //This calls findUserOrDie which will retrieve the key param or api token header, or the workflow token header. - User apiTokenUser = findAPITokenUser(getRequestUser(crc)); - gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, apiTokenUser); + gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, getUser(crc)); guestbookResponseService.save(gbr); MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, df); mdcLogService.logEntry(entry); @@ -283,13 +280,12 @@ public Response datafile(@Context ContainerRequestContext crc, @PathParam("fileI // (nobody should ever be using this API on a harvested DataFile)! } - // This will throw a ForbiddenException if access isn't authorized: - checkAuthorization(getRequestUser(crc), df); + // This will throw a ForbiddenException if access isn't authorized: + checkAuthorization(crc, df); if (gbrecs != true && df.isReleased()){ // Write Guestbook record if not done previously and file is released - User apiTokenUser = findAPITokenUser(getRequestUser(crc)); - gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, apiTokenUser); + gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, getUser(crc)); } DownloadInfo dInfo = new DownloadInfo(df); @@ -624,7 +620,7 @@ public DownloadInstance downloadAuxiliaryFile(@Context ContainerRequestContext c // as defined for the DataFile itself), and will throw a ForbiddenException // if access is denied: if (!publiclyAvailable) { - checkAuthorization(getRequestUser(crc), df); + checkAuthorization(crc, df); } return downloadInstance; @@ -644,7 +640,7 @@ public DownloadInstance downloadAuxiliaryFile(@Context ContainerRequestContext c public Response postDownloadDatafiles(@Context ContainerRequestContext crc, String fileIds, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { - return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, null); + return downloadDatafiles(crc, fileIds, gbrecs, uriInfo, headers, response, null); } @GET @@ -665,7 +661,7 @@ public Response downloadAllFromLatest(@Context ContainerRequestContext crc, @Pat // We don't want downloads from Draft versions to be counted, // so we are setting the gbrecs (aka "do not write guestbook response") // variable accordingly: - return downloadDatafiles(getRequestUser(crc), fileIds, true, uriInfo, headers, response, "draft"); + return downloadDatafiles(crc, fileIds, true, uriInfo, headers, response, "draft"); } } @@ -686,7 +682,7 @@ public Response downloadAllFromLatest(@Context ContainerRequestContext crc, @Pat } String fileIds = getFileIdsAsCommaSeparated(latest.getFileMetadatas()); - return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, latest.getFriendlyVersionNumber()); + return downloadDatafiles(crc, fileIds, gbrecs, uriInfo, headers, response, latest.getFriendlyVersionNumber()); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -736,7 +732,7 @@ public Command handleLatestPublished() { if (dsv.isDraft()) { gbrecs = true; } - return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, dsv.getFriendlyVersionNumber().toLowerCase()); + return downloadDatafiles(crc, fileIds, gbrecs, uriInfo, headers, response, dsv.getFriendlyVersionNumber().toLowerCase()); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -777,10 +773,10 @@ private String generateMultiFileBundleName(Dataset dataset, String versionTag) { @Path("datafiles/{fileIds}") @Produces({"application/zip"}) public Response datafiles(@Context ContainerRequestContext crc, @PathParam("fileIds") String fileIds, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { - return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, null); + return downloadDatafiles(crc, fileIds, gbrecs, uriInfo, headers, response, null); } - private Response downloadDatafiles(User user, String rawFileIds, boolean donotwriteGBResponse, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response, String versionTag) throws WebApplicationException /* throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { + private Response downloadDatafiles(ContainerRequestContext crc, String rawFileIds, boolean donotwriteGBResponse, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response, String versionTag) throws WebApplicationException /* throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { final long zipDownloadSizeLimit = systemConfig.getZipDownloadLimit(); logger.fine("setting zip download size limit to " + zipDownloadSizeLimit + " bytes."); @@ -801,8 +797,8 @@ private Response downloadDatafiles(User user, String rawFileIds, boolean donotwr String customZipServiceUrl = settingsService.getValueForKey(SettingsServiceBean.Key.CustomZipDownloadServiceUrl); boolean useCustomZipService = customZipServiceUrl != null; - - User apiTokenUser = findAPITokenUser(user); //for use in adding gb records if necessary + + User user = getUser(crc); Boolean getOrig = false; for (String key : uriInfo.getQueryParameters().keySet()) { @@ -815,7 +811,7 @@ private Response downloadDatafiles(User user, String rawFileIds, boolean donotwr if (useCustomZipService) { URI redirect_uri = null; try { - redirect_uri = handleCustomZipDownload(user, customZipServiceUrl, fileIds, apiTokenUser, uriInfo, headers, donotwriteGBResponse, true); + redirect_uri = handleCustomZipDownload(user, customZipServiceUrl, fileIds, uriInfo, headers, donotwriteGBResponse, true); } catch (WebApplicationException wae) { throw wae; } @@ -860,7 +856,7 @@ public void write(OutputStream os) throws IOException, logger.fine("adding datafile (id=" + file.getId() + ") to the download list of the ZippedDownloadInstance."); //downloadInstance.addDataFile(file); if (donotwriteGBResponse != true && file.isReleased()){ - GuestbookResponse gbr = guestbookResponseService.initAPIGuestbookResponse(file.getOwner(), file, session, apiTokenUser); + GuestbookResponse gbr = guestbookResponseService.initAPIGuestbookResponse(file.getOwner(), file, session, user); guestbookResponseService.save(gbr); MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, file); mdcLogService.logEntry(entry); @@ -1735,15 +1731,22 @@ public Response getUserPermissionsOnFile(@Context ContainerRequestContext crc, @ } // checkAuthorization is a convenience method; it calls the boolean method - // isAccessAuthorized(), the actual workhorse, tand throws a 403 exception if not. - - private void checkAuthorization(User user, DataFile df) throws WebApplicationException { - + // isAccessAuthorized(), the actual workhorse, and throws a 403 exception if not. + private void checkAuthorization(ContainerRequestContext crc, DataFile df) throws WebApplicationException { + User user = getUser(crc); if (!isAccessAuthorized(user, df)) { throw new ForbiddenException(); } } - + private User getUser(ContainerRequestContext crc) { + User user = getRequestUser(crc); + // CompoundAuthMechanism should find the user by API Key/Token, Workflow, etc. And for SPA the Bearer Token + // For JSF check if CompoundAuthMechanism couldn't find the user then try to get it from the session + if (session!=null && user instanceof GuestUser) { + user = session.getUser(); + } + return user; + } private boolean isAccessAuthorized(User requestUser, DataFile df) { // First, check if the file belongs to a released Dataset version: @@ -1818,8 +1821,6 @@ private boolean isAccessAuthorized(User requestUser, DataFile df) { } } } - - //The one case where we don't need to check permissions if (!restricted && !embargoed && !retentionExpired && published) { @@ -1828,50 +1829,11 @@ private boolean isAccessAuthorized(User requestUser, DataFile df) { // be handled below) return true; } - - //For permissions check decide if we have a session user, or an API user - User sessionUser = null; - + /** * Authentication/authorization: */ - User apiUser = requestUser; - - /* - * If API user is not authenticated, and a session user exists, we use that. - * If the API user indicates a GuestUser, we will use that if there's no session. - * - * This is currently the only API call that supports sessions. If the rest of - * the API is opened up, the custom logic here wouldn't be needed. - */ - - if ((apiUser instanceof GuestUser) && session != null) { - if (session.getUser() != null) { - sessionUser = session.getUser(); - apiUser = null; - //Fine logging - if (!session.getUser().isAuthenticated()) { - logger.fine("User associated with the session is not an authenticated user."); - if (session.getUser() instanceof PrivateUrlUser) { - logger.fine("User associated with the session is a PrivateUrlUser user."); - } - if (session.getUser() instanceof GuestUser) { - logger.fine("User associated with the session is indeed a guest user."); - } - } - } else { - logger.fine("No user associated with the session."); - } - } else { - logger.fine("Session is null."); - } - //If we don't have a user, nothing more to do. (Note session could have returned GuestUser) - if (sessionUser == null && apiUser == null) { - logger.warning("Unable to find a user via session or with a token."); - return false; - } - /* * Since published and not restricted/embargoed is handled above, the main split * now is whether it is published or not. If it's published, the only case left @@ -1879,13 +1841,8 @@ private boolean isAccessAuthorized(User requestUser, DataFile df) { * and not restricted/embargoed both get handled the same way. */ - DataverseRequest dvr = null; - if (apiUser != null) { - dvr = createDataverseRequest(apiUser); - } else { - // used in JSF context, user may be Guest - dvr = dvRequestService.getDataverseRequest(); - } + DataverseRequest dvr = createDataverseRequest(requestUser); + if (!published) { // and restricted or embargoed (implied by earlier processing) // If the file is not published, they can still download the file, if the user // has the permission to view unpublished versions: @@ -1895,7 +1852,7 @@ private boolean isAccessAuthorized(User requestUser, DataFile df) { // it's not unthinkable, that a GuestUser could be given // the ViewUnpublished permission! logger.log(Level.FINE, - "Session-based auth: user {0} has access rights on the non-restricted, unpublished datafile.", + "auth: user {0} has access rights on the non-restricted, unpublished datafile.", dvr.getUser().getIdentifier()); return true; } @@ -1905,35 +1862,11 @@ private boolean isAccessAuthorized(User requestUser, DataFile df) { return true; } } - if (sessionUser != null) { - logger.log(Level.FINE, "Session-based auth: user {0} has NO access rights on the requested datafile.", sessionUser.getIdentifier()); - } - - if (apiUser != null) { - logger.log(Level.FINE, "Token-based auth: user {0} has NO access rights on the requested datafile.", apiUser.getIdentifier()); - } - return false; - } - - - private User findAPITokenUser(User requestUser) { - User apiTokenUser = requestUser; - /* - * The idea here is to not let a guest user coming from the request (which - * happens when there is no key/token, and which we want if there's no session) - * from overriding an authenticated session user. - */ - if(apiTokenUser instanceof GuestUser) { - if(session!=null && session.getUser()!=null) { - //The apiTokenUser, if set, will override the sessionUser in permissions calcs, so set it to null if we have a session user - apiTokenUser=null; - } - } - return apiTokenUser; + return false; } - private URI handleCustomZipDownload(User user, String customZipServiceUrl, String fileIds, User apiTokenUser, UriInfo uriInfo, HttpHeaders headers, boolean donotwriteGBResponse, boolean orig) throws WebApplicationException { + private URI handleCustomZipDownload(User user, String customZipServiceUrl, String fileIds, UriInfo uriInfo, HttpHeaders headers, boolean donotwriteGBResponse, boolean orig) throws WebApplicationException { String zipServiceKey = null; Timestamp timestamp = null; @@ -1962,7 +1895,7 @@ private URI handleCustomZipDownload(User user, String customZipServiceUrl, Strin if (isAccessAuthorized(user, file)) { logger.fine("adding datafile (id=" + file.getId() + ") to the download list of the ZippedDownloadInstance."); if (donotwriteGBResponse != true && file.isReleased()) { - GuestbookResponse gbr = guestbookResponseService.initAPIGuestbookResponse(file.getOwner(), file, session, apiTokenUser); + GuestbookResponse gbr = guestbookResponseService.initAPIGuestbookResponse(file.getOwner(), file, session, user); guestbookResponseService.save(gbr); MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, file); mdcLogService.logEntry(entry); From a33728d4553fed345226b8c9a69c78fb03fab097 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 25 Sep 2025 09:46:48 -0400 Subject: [PATCH 039/578] add release note --- .../11740-api-file-download-with-bearer-token.md | 7 +++++++ src/main/java/edu/harvard/iq/dataverse/api/Access.java | 10 +++++----- 2 files changed, 12 insertions(+), 5 deletions(-) create mode 100644 doc/release-notes/11740-api-file-download-with-bearer-token.md diff --git a/doc/release-notes/11740-api-file-download-with-bearer-token.md b/doc/release-notes/11740-api-file-download-with-bearer-token.md new file mode 100644 index 00000000000..651ae4040d6 --- /dev/null +++ b/doc/release-notes/11740-api-file-download-with-bearer-token.md @@ -0,0 +1,7 @@ +## Bug / Not Bug in Dataverse. Bug is in SPA Frontend + +Cleaned up Access APIs to localize getting user from session for JSF backward compatibility + +This bug requires a front end fix to send the Bearer Token in the API call. + +See: #11740 diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Access.java b/src/main/java/edu/harvard/iq/dataverse/api/Access.java index 10ed320ec76..cadd758a3ac 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Access.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Access.java @@ -189,7 +189,7 @@ public BundleDownloadInstance datafileBundle(@Context ContainerRequestContext cr if (gbrecs != true && df.isReleased()){ // Write Guestbook record if not done previously and file is released - gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, getUser(crc)); + gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, getRequestor(crc)); guestbookResponseService.save(gbr); MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, df); mdcLogService.logEntry(entry); @@ -285,7 +285,7 @@ public Response datafile(@Context ContainerRequestContext crc, @PathParam("fileI if (gbrecs != true && df.isReleased()){ // Write Guestbook record if not done previously and file is released - gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, getUser(crc)); + gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, getRequestor(crc)); } DownloadInfo dInfo = new DownloadInfo(df); @@ -798,7 +798,7 @@ private Response downloadDatafiles(ContainerRequestContext crc, String rawFileId String customZipServiceUrl = settingsService.getValueForKey(SettingsServiceBean.Key.CustomZipDownloadServiceUrl); boolean useCustomZipService = customZipServiceUrl != null; - User user = getUser(crc); + User user = getRequestor(crc); Boolean getOrig = false; for (String key : uriInfo.getQueryParameters().keySet()) { @@ -1733,12 +1733,12 @@ public Response getUserPermissionsOnFile(@Context ContainerRequestContext crc, @ // checkAuthorization is a convenience method; it calls the boolean method // isAccessAuthorized(), the actual workhorse, and throws a 403 exception if not. private void checkAuthorization(ContainerRequestContext crc, DataFile df) throws WebApplicationException { - User user = getUser(crc); + User user = getRequestor(crc); if (!isAccessAuthorized(user, df)) { throw new ForbiddenException(); } } - private User getUser(ContainerRequestContext crc) { + private User getRequestor(ContainerRequestContext crc) { User user = getRequestUser(crc); // CompoundAuthMechanism should find the user by API Key/Token, Workflow, etc. And for SPA the Bearer Token // For JSF check if CompoundAuthMechanism couldn't find the user then try to get it from the session From 49a7f9454edb7a5c5df097481d0475fb921f98af Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Fri, 26 Sep 2025 09:23:02 -0400 Subject: [PATCH 040/578] refactor guestbook --- .../GuestbookResponseServiceBean.java | 31 +++++-------------- 1 file changed, 7 insertions(+), 24 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java index 6c043b78941..2393d72dd2a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java @@ -762,30 +762,13 @@ private void initCustomQuestions(GuestbookResponse guestbookResponse, Dataset da } } - private void setUserDefaultResponses(GuestbookResponse guestbookResponse, DataverseSession session, User userIn) { - User user; - User sessionUser = session.getUser(); - - if (userIn != null){ - user = userIn; - } else{ - user = sessionUser; - } - - if (user != null) { - guestbookResponse.setEmail(getUserEMail(user)); - guestbookResponse.setName(getUserName(user)); - guestbookResponse.setInstitution(getUserInstitution(user)); - guestbookResponse.setPosition(getUserPosition(user)); - guestbookResponse.setAuthenticatedUser(getAuthenticatedUser(user)); - } else { - guestbookResponse.setEmail(""); - guestbookResponse.setName(""); - guestbookResponse.setInstitution(""); - guestbookResponse.setPosition(""); - guestbookResponse.setAuthenticatedUser(null); - } - guestbookResponse.setSessionId(session.toString()); + private void setUserDefaultResponses(GuestbookResponse guestbookResponse, DataverseSession session, User user) { + guestbookResponse.setEmail(getUserEMail(user)); + guestbookResponse.setName(getUserName(user)); + guestbookResponse.setInstitution(getUserInstitution(user)); + guestbookResponse.setPosition(getUserPosition(user)); + guestbookResponse.setAuthenticatedUser(getAuthenticatedUser(user)); + guestbookResponse.setSessionId(session != null ? session.toString() : ""); } private void setUserDefaultResponses(GuestbookResponse guestbookResponse, DataverseSession session) { From 9cdb4f6396612fbf58bdb9451f3c8ad6f3e6a0f7 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Fri, 3 Oct 2025 10:57:37 -0400 Subject: [PATCH 041/578] updates to send context, use as namespace, use relationType, DASH-NRS --- .../LDNAnnounceDatasetVersionStep.java | 33 ++++++++++++++----- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java index abe9f26f98d..deb7ccfbb9f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -176,20 +176,26 @@ JsonArray getObjects(WorkflowContext ctxt, Map fields) { private JsonObject getRelationshipObject(DatasetFieldType dft, JsonValue jval, Dataset d, Map localContext) { - String id = getBestId(dft, jval); - return Json.createObjectBuilder().add("object", id).add("relationship", dft.getJsonLDTerm().getUrl()) - .add("subject", d.getGlobalId().asURL().toString()).add("id", "urn:uuid:" + UUID.randomUUID().toString()).add("type","Relationship").build(); + String[] answers = getBestIdAndType(dft, jval); + String id = answers[0]; + String type = answers[1]; + return Json.createObjectBuilder().add("as:object", id).add("as:relationship", type) + .add("as:subject", d.getGlobalId().asURL().toString()).add("id", "urn:uuid:" + UUID.randomUUID().toString()).add("type","Relationship").build(); } HttpPost buildAnnouncement(Dataset d, JsonObject rel, JsonObject target) throws URISyntaxException { JsonObjectBuilder job = Json.createObjectBuilder(); - JsonArrayBuilder context = Json.createArrayBuilder().add("https://purl.org/coar/notify") - .add("https://www.w3.org/ns/activitystreams"); + JsonArrayBuilder context = Json.createArrayBuilder() + .add("https://www.w3.org/ns/activitystreams") + .add("https://coar-notify.net"); job.add("@context", context); job.add("id", "urn:uuid:" + UUID.randomUUID().toString()); job.add("actor", Json.createObjectBuilder().add("id", SystemConfig.getDataverseSiteUrlStatic()) .add("name", BrandingUtil.getInstallationBrandName()).add("type", "Service")); + JsonObjectBuilder coarContextBuilder = Json.createObjectBuilder(); + coarContextBuilder.add("id", rel.getString("as:object")); + job.add("context", coarContextBuilder.build()); job.add("object", rel); job.add("origin", Json.createObjectBuilder().add("id", SystemConfig.getDataverseSiteUrlStatic()) .add("inbox", SystemConfig.getDataverseSiteUrlStatic() + "/api/inbox").add("type", "Service")); @@ -205,10 +211,12 @@ HttpPost buildAnnouncement(Dataset d, JsonObject rel, JsonObject target) throws return annPost; } - private String getBestId(DatasetFieldType dft, JsonValue jv) { + private String[] getBestIdAndType(DatasetFieldType dft, JsonValue jv) { + + String type = "https://purl.org/datacite/ontology#isSupplementTo"; // Primitive value if (jv instanceof JsonString) { - return ((JsonString) jv).getString(); + return new String[] { ((JsonString) jv).getString(), type }; } // Compound - apply type specific logic to get best Id JsonObject jo = jv.asJsonObject(); @@ -218,6 +226,7 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { JsonLDTerm publicationIDType = null; JsonLDTerm publicationIDNumber = null; JsonLDTerm publicationURL = null; + JsonLDTerm publicationRelationType = null; Collection childTypes = dft.getChildDatasetFieldTypes(); for (DatasetFieldType cdft : childTypes) { @@ -231,6 +240,8 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { case "publicationIDNumber": publicationIDNumber = cdft.getJsonLDTerm(); break; + case "publicationRelationType": + publicationRelationType = cdft.getJsonLDTerm(); } } if (jo.containsKey(publicationURL.getLabel())) { @@ -250,7 +261,7 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { id = "https://doi.org/" + number; } break; - case "DASH-URN": + case "DASH-NRS": if (number.startsWith("http")) { id = number; } @@ -258,6 +269,10 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { } } } + if(jo.containsKey(publicationRelationType.getLabel())) { + type = jo.getString(publicationRelationType.getLabel()); + type = "https://purl.org/datacite/ontology#" + type.substring(0,1).toLowerCase() + type.substring(1); + } break; default: Collection childDFTs = dft.getChildDatasetFieldTypes(); @@ -300,7 +315,7 @@ private String getBestId(DatasetFieldType dft, JsonValue jv) { } id = jo.getString(jo.keySet().iterator().next()); } - return id; + return new String[] {id, type}; } String process(String template, Map values) { From 0ffbb8e9aba9d99ba55f87fc565433e4a8c1459a Mon Sep 17 00:00:00 2001 From: qqmyers Date: Mon, 6 Oct 2025 13:59:26 -0400 Subject: [PATCH 042/578] doc updates --- doc/release-notes/10490-COAR-Notify.md | 19 +++++++++++++++++++ .../source/developers/workflows.rst | 5 +++-- 2 files changed, 22 insertions(+), 2 deletions(-) create mode 100644 doc/release-notes/10490-COAR-Notify.md diff --git a/doc/release-notes/10490-COAR-Notify.md b/doc/release-notes/10490-COAR-Notify.md new file mode 100644 index 00000000000..934288d4d4c --- /dev/null +++ b/doc/release-notes/10490-COAR-Notify.md @@ -0,0 +1,19 @@ +### Support for COAR Notify Relationship Announcement + +Dataverse now supports sending and recieving [Linked Data Notification ](https://www.w3.org/TR/ldn/) messages involved in the +[COAR Notify Relationship Announcement Workflow](https://coar-notify.net/catalogue/workflows/repository-relationship-repository/). + +Dataverse can send messages to configured repositories announcing that a dataset has a related publication (as defined in the dataset metadata). This may be done automatically upon publication or triggered manually by a superuser. The receiving repository may do anything with the message, with the default expectation being that the repository will create a backlink from the publication to the dataset (assuming the publication exists in the repository, admins agree the link makes sense, etc.) + +Conversely, Dataverse can recieve notices from other configured repositories announcing relationships between their publications and datasets. If the referenced dataset exists in the Dataverse instance, a notification will be sent to users who can publish the dataset. They can then decide whether to create a backlink to the publication in the dataset metadata. + +(Earlier releases of Dataverse had experimental support in this area that was based on message formats defined prior to finalization of the COAR Notify specification for relationship announcements.) + +Configuration for sending messages involves specifying the +:LDNTarget and :LDNAnnounceRequiredFields + +Configuration to receive messages involves specifying the +:LDNMessageHosts + +(FWIW: These settings are not new) + diff --git a/doc/sphinx-guides/source/developers/workflows.rst b/doc/sphinx-guides/source/developers/workflows.rst index 38ca6f4e141..a7f6c8a044c 100644 --- a/doc/sphinx-guides/source/developers/workflows.rst +++ b/doc/sphinx-guides/source/developers/workflows.rst @@ -205,13 +205,13 @@ Note - the example step includes two settings required for any archiver, three ( ldnannounce +++++++++++ -An experimental step that sends a Linked Data Notification (LDN) message to a specific LDN Inbox announcing the publication/availability of a dataset meeting certain criteria. +A step that sends a `Linked Data Notification (LDN)`_ message to a specific LDN Inbox announcing a relationship between an newly published/available dataset with a relationship to an external resource (e.g. one managed by the recipient). The two parameters are * ``:LDNAnnounceRequiredFields`` - a list of metadata fields that must exist to trigger the message. Currently, the message also includes the values for these fields but future versions may only send the dataset's persistent identifier (making the receiver responsible for making a call-back to get any metadata). * ``:LDNTarget`` - a JSON object containing an ``inbox`` key whose value is the URL of the target LDN inbox to which messages should be sent, e.g. ``{"id": "https://dashv7-dev.lib.harvard.edu","inbox": "https://dashv7-api-dev.lib.harvard.edu/server/ldn/inbox","type": "Service"}`` ). -The supported message format is desribed by `our preliminary specification `_. The format is expected to change in the near future to match the standard for relationship announcements being developed as part of `the COAR Notify Project `_. +The message format is defined by the `COAR Notify Relationship Announcement `_ standard. .. code:: json @@ -224,6 +224,7 @@ The supported message format is desribed by `our preliminary specification Date: Mon, 6 Oct 2025 13:59:35 -0400 Subject: [PATCH 043/578] note some ToDos --- .../internalspi/LDNAnnounceDatasetVersionStep.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java index deb7ccfbb9f..7d2aa166461 100644 --- a/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java +++ b/src/main/java/edu/harvard/iq/dataverse/workflow/internalspi/LDNAnnounceDatasetVersionStep.java @@ -56,6 +56,8 @@ public class LDNAnnounceDatasetVersionStep implements WorkflowStep { private static final Logger logger = Logger.getLogger(LDNAnnounceDatasetVersionStep.class.getName()); + //ToDo - not required fields at this point - each results in a message, so a) change to LDNAnnounceFields, and b) consider settings + // connecting field and targets (only DB settings are supported in workflows at present) private static final String REQUIRED_FIELDS = ":LDNAnnounceRequiredFields"; private static final String LDN_TARGET = ":LDNTarget"; private static final String RELATED_PUBLICATION = "publication"; @@ -274,7 +276,10 @@ private String[] getBestIdAndType(DatasetFieldType dft, JsonValue jv) { type = "https://purl.org/datacite/ontology#" + type.substring(0,1).toLowerCase() + type.substring(1); } break; - default: + default: + //ToDo - handle primary field + //ToDo - handle "Identifier" vs "IdentifierType" + //ToDo - check for URL form Collection childDFTs = dft.getChildDatasetFieldTypes(); // Loop through child fields and select one // The order of preference is for a field with URL in the name, followed by one From 26e0ee45fc1f6c3f29d077d79201d3075660942a Mon Sep 17 00:00:00 2001 From: qqmyers Date: Tue, 7 Oct 2025 13:11:41 -0400 Subject: [PATCH 044/578] New format has strings, not objects with @id --- .../java/edu/harvard/iq/dataverse/api/LDNInbox.java | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java index b9bddb02cfe..6125027f41f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/LDNInbox.java @@ -119,8 +119,8 @@ public Response acceptMessage(String body) { if (new JsonLDTerm(activityStreams, "Relationship").getUrl().equals(msgObject.getString("@type"))) { // We have a relationship message - need to get the subject and object and // relationship type - String subjectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "subject").getUrl()).getString("@id"); - String objectId = msgObject.getJsonObject(new JsonLDTerm(activityStreams, "object").getUrl()).getString("@id"); + String subjectId = msgObject.getString(new JsonLDTerm(activityStreams, "subject").getUrl()); + String objectId = msgObject.getString(new JsonLDTerm(activityStreams, "object").getUrl()); // Best-effort to get name by following redirects and looing for a 'name' field in the returned json try (CloseableHttpClient client = HttpClients.createDefault()) { logger.info("Getting " + subjectId); @@ -156,7 +156,7 @@ public Response acceptMessage(String body) { logger.info(e.getLocalizedMessage()); } String relationshipId = msgObject - .getJsonObject(new JsonLDTerm(activityStreams, "relationship").getUrl()).getString("@id"); + .getString(new JsonLDTerm(activityStreams, "relationship").getUrl()); if (subjectId != null && objectId != null && relationshipId != null) { // Strip the URL part from a relationship ID/URL assuming a usable label exists // after a # char. Default is to use the whole URI. @@ -178,11 +178,7 @@ public Response acceptMessage(String body) { citingResourceBuilder.add("@type",itemType.substring(0,1).toUpperCase() + itemType.substring(1)); } JsonObject citingResource = citingResourceBuilder.build(); - StringWriter sw = new StringWriter(128); - try (JsonWriter jw = Json.createWriter(sw)) { - jw.write(citingResource); - } - String jsonstring = sw.toString(); + String jsonstring = JsonUtil.prettyPrint(citingResource); logger.info("Storing: " + jsonstring); //Set ras = roleService.rolesAssignments(dataset); From 85c45a22dc1157ac31f9616d8e87724a67e822b2 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Fri, 10 Oct 2025 11:47:35 -0400 Subject: [PATCH 045/578] add to native api doc --- doc/sphinx-guides/source/api/native-api.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index 19a4d79c5ae..4a8b43e0ea7 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -1464,6 +1464,8 @@ Get JSON Representation of a Dataset .. note:: Datasets can be accessed using persistent identifiers. This is done by passing the constant ``:persistentId`` where the numeric id of the dataset is expected, and then passing the actual persistent id as a query parameter with the name ``persistentId``. +If a user with EditDataset permissions wants to ignore the setting ``ExcludeEmailFromExport`` in order to see the contact email, they must include the ``ignoreSettingExcludeEmailFromExport`` query parameter (Required by SPA). + Example: Getting the dataset whose DOI is *10.5072/FK2/J8SJZB*: .. code-block:: bash @@ -1471,13 +1473,13 @@ Example: Getting the dataset whose DOI is *10.5072/FK2/J8SJZB*: export SERVER_URL=https://demo.dataverse.org export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/J8SJZB - curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/datasets/:persistentId/?persistentId=$PERSISTENT_IDENTIFIER" + curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/datasets/:persistentId/?persistentId=$PERSISTENT_IDENTIFIER&ignoreSettingExcludeEmailFromExport" The fully expanded example above (without environment variables) looks like this: .. code-block:: bash - curl -H "X-Dataverse-key:$API_TOKEN" "https://demo.dataverse.org/api/datasets/:persistentId/?persistentId=doi:10.5072/FK2/J8SJZB" + curl -H "X-Dataverse-key:$API_TOKEN" "https://demo.dataverse.org/api/datasets/:persistentId/?persistentId=doi:10.5072/FK2/J8SJZB&ignoreSettingExcludeEmailFromExport" Getting its draft version: From ddba5fddf7b05f4682af35a5f4e382549ea78fde Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Tue, 21 Oct 2025 10:25:30 -0400 Subject: [PATCH 046/578] limit notifications role agnment list --- ...ect-roles-listed-in-assignrole-notifications.md | 4 ++++ .../iq/dataverse/ManagePermissionsPage.java | 14 +++++++++----- .../providers/builtin/DataverseUserPage.java | 14 ++++++++++---- 3 files changed, 23 insertions(+), 9 deletions(-) create mode 100644 doc/release-notes/11773-incorrect-roles-listed-in-assignrole-notifications.md diff --git a/doc/release-notes/11773-incorrect-roles-listed-in-assignrole-notifications.md b/doc/release-notes/11773-incorrect-roles-listed-in-assignrole-notifications.md new file mode 100644 index 00000000000..d425c8551a5 --- /dev/null +++ b/doc/release-notes/11773-incorrect-roles-listed-in-assignrole-notifications.md @@ -0,0 +1,4 @@ +This release changes the text in assign role notifications to list only the role being assigned that generated the specific notification. +The previous implementation listed all the roles associated with the dataset in each notification. + +See also [the guides](https://dataverse-guide--11664.org.readthedocs.build/en/11664/user/account.html#notifications) and #11773. diff --git a/src/main/java/edu/harvard/iq/dataverse/ManagePermissionsPage.java b/src/main/java/edu/harvard/iq/dataverse/ManagePermissionsPage.java index 0e277c5aa32..e853c55d77f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/ManagePermissionsPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/ManagePermissionsPage.java @@ -193,7 +193,7 @@ private void revokeRole(RoleAssignment ra) { commandEngine.submit(new RevokeRoleCommand(ra, dvRequestService.getDataverseRequest())); JsfHelper.addSuccessMessage(BundleUtil.getStringFromBundle("permission.roleWasRemoved", Arrays.asList(ra.getRole().getName(), roleAssigneeService.getRoleAssignee(ra.getAssigneeIdentifier()).getDisplayInfo().getTitle()))); RoleAssignee assignee = roleAssigneeService.getRoleAssignee(ra.getAssigneeIdentifier()); - notifyRoleChange(assignee, UserNotification.Type.REVOKEROLE); + notifyRoleChange(assignee, UserNotification.Type.REVOKEROLE, ra.getRole()); } catch (PermissionException ex) { JH.addMessage(FacesMessage.SEVERITY_ERROR, BundleUtil.getStringFromBundle("permission.roleNotAbleToBeRemoved"), BundleUtil.getStringFromBundle("permission.permissionsMissing", Arrays.asList(ex.getRequiredPermissions().toString()))); } catch (CommandException ex) { @@ -504,17 +504,21 @@ public void assignRole(ActionEvent evt) { * Will notify all members of a group. * @param ra The {@code RoleAssignee} to be notified. * @param type The type of notification. + * @param r The {@code DataverseRole} associated with the change */ - private void notifyRoleChange(RoleAssignee ra, UserNotification.Type type) { + private void notifyRoleChange(RoleAssignee ra, UserNotification.Type type, DataverseRole r) { + String additionalInfo = r != null ? String.format("{ roleId: %d, roleName: %s }", r.getId(), r.getName()) : null; if (ra instanceof AuthenticatedUser) { - userNotificationService.sendNotification((AuthenticatedUser) ra, new Timestamp(new Date().getTime()), type, dvObject.getId()); + userNotificationService.sendNotification((AuthenticatedUser) ra, new Timestamp(new Date().getTime()), type, + dvObject.getId(), null, null, false, additionalInfo); } else if (ra instanceof ExplicitGroup) { ExplicitGroup eg = (ExplicitGroup) ra; Set explicitGroupMembers = eg.getContainedRoleAssgineeIdentifiers(); for (String id : explicitGroupMembers) { RoleAssignee explicitGroupMember = roleAssigneeService.getRoleAssignee(id); if (explicitGroupMember instanceof AuthenticatedUser) { - userNotificationService.sendNotification((AuthenticatedUser) explicitGroupMember, new Timestamp(new Date().getTime()), type, dvObject.getId()); + userNotificationService.sendNotification((AuthenticatedUser) explicitGroupMember, new Timestamp(new Date().getTime()), type, + dvObject.getId(), null, null, false, additionalInfo); } } } @@ -532,7 +536,7 @@ private void assignRole(RoleAssignee ra, DataverseRole r) { JsfHelper.addSuccessMessage(BundleUtil.getStringFromBundle("permission.roleAssignedToFor", args)); // don't notify if role = file downloader and object is not released if (!(r.getAlias().equals(DataverseRole.FILE_DOWNLOADER) && !dvObject.isReleased()) ){ - notifyRoleChange(ra, UserNotification.Type.ASSIGNROLE); + notifyRoleChange(ra, UserNotification.Type.ASSIGNROLE, r); } } catch (PermissionException ex) { diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java index cda94d25060..ca89a3e63fc 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java @@ -69,6 +69,7 @@ import edu.harvard.iq.dataverse.authorization.providers.oauth2.impl.OrcidOAuth2AP; import java.io.IOException; import org.apache.commons.lang3.StringUtils; +import org.json.JSONObject; import org.primefaces.event.TabChangeEvent; /** @@ -453,8 +454,13 @@ public void onTabChange(TabChangeEvent event) { } - private String getRoleStringFromUser(AuthenticatedUser au, DvObject dvObj) { + private String getRoleStringFromUser(AuthenticatedUser au, DvObject dvObj, String additionalInfo) { // Find user's role(s) for given dataverse/dataset + if (additionalInfo != null && additionalInfo.contains("roleName:")) { + JSONObject jsonObject = new JSONObject(additionalInfo); + return jsonObject.getString("roleName"); + } + Set roles = permissionService.assignmentsFor(au, dvObj); List roleNames = new ArrayList<>(); @@ -481,16 +487,16 @@ public void displayNotification() { // Can either be a dataverse or dataset, so search both Dataverse dataverse = dataverseService.find(userNotification.getObjectId()); if (dataverse != null) { - userNotification.setRoleString(this.getRoleStringFromUser(this.getCurrentUser(), dataverse )); + userNotification.setRoleString(this.getRoleStringFromUser(this.getCurrentUser(), dataverse, userNotification.getAdditionalInfo())); userNotification.setTheObject(dataverse); } else { Dataset dataset = datasetService.find(userNotification.getObjectId()); if (dataset != null){ - userNotification.setRoleString(this.getRoleStringFromUser(this.getCurrentUser(), dataset )); + userNotification.setRoleString(this.getRoleStringFromUser(this.getCurrentUser(), dataset, userNotification.getAdditionalInfo())); userNotification.setTheObject(dataset); } else { DataFile datafile = fileService.find(userNotification.getObjectId()); - userNotification.setRoleString(this.getRoleStringFromUser(this.getCurrentUser(), datafile )); + userNotification.setRoleString(this.getRoleStringFromUser(this.getCurrentUser(), datafile, userNotification.getAdditionalInfo())); userNotification.setTheObject(datafile); } } From e9f6b71ddeadf987da20b0d6f2601133cdaa6dbe Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 23 Oct 2025 10:11:47 -0400 Subject: [PATCH 047/578] Hide SPA OIDC Providers from JSF login screen --- .../edu/harvard/iq/dataverse/LoginPage.java | 8 +++- .../authorization/AuthenticationProvider.java | 3 +- .../oauth2/oidc/OIDCAuthProvider.java | 9 +++-- .../OIDCAuthenticationProviderFactory.java | 6 ++- .../iq/dataverse/settings/JvmSettings.java | 1 + src/main/webapp/loginpage.xhtml | 6 +-- .../edu/harvard/iq/dataverse/api/AdminIT.java | 39 +++++++++++++++++++ .../edu/harvard/iq/dataverse/api/UtilIT.java | 8 ++++ 8 files changed, 70 insertions(+), 10 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/LoginPage.java b/src/main/java/edu/harvard/iq/dataverse/LoginPage.java index ecc249cee87..3580d3b596b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/LoginPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/LoginPage.java @@ -135,7 +135,13 @@ public List listCredentialsAuthenticationProv * Retrieve information about all enabled identity providers in a sorted order to be displayed to the user. * @return list of display information for each provider */ + public List listAuthenticationProvidersJSF() { + return listAuthenticationProviders(false); + } public List listAuthenticationProviders() { + return listAuthenticationProviders(true); + } + public List listAuthenticationProviders(boolean ignoreBlocked) { List infos = new LinkedList<>(); List idps = new ArrayList<>(authSvc.getAuthenticationProviders()); @@ -143,7 +149,7 @@ public List listAuthenticationProviders() { Collections.sort(idps, Comparator.comparing(AuthenticationProvider::getOrder).thenComparing(AuthenticationProvider::getId)); for (AuthenticationProvider idp : idps) { - if (idp != null) { + if (idp != null && (ignoreBlocked || !idp.isJsfBlocked())) { infos.add(idp.getInfo()); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java index 8dfe7ca86b3..3a588d15ea8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java @@ -18,7 +18,7 @@ * "authenticationProvider.name." + "ship" -> * (c) Bundle.properties entry: "authenticationProvider.name.shib=Shibboleth" * - * {@code AuthenticationPrvider}s are normally registered at startup in {@link AuthenticationServiceBean#startup()}. + * {@code AuthenticationProvider}s are normally registered at startup in {@link AuthenticationServiceBean#startup()}. * * @author michael */ @@ -33,6 +33,7 @@ public interface AuthenticationProvider { default boolean isUserInfoUpdateAllowed() { return false; }; default boolean isUserDeletionAllowed() { return false; }; default boolean isOAuthProvider() { return false; }; + default boolean isJsfBlocked() { return false; }; diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java index 3e62598ee79..d9c69920cb3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java @@ -67,6 +67,7 @@ public class OIDCAuthProvider extends AbstractOAuth2AuthenticationProvider { final OIDCProviderMetadata idpMetadata; final boolean pkceEnabled; final CodeChallengeMethod pkceMethod; + final boolean jsfBlocked; /** * Using PKCE, we create and send a special {@link CodeVerifier}. This contains a secret @@ -80,8 +81,8 @@ public class OIDCAuthProvider extends AbstractOAuth2AuthenticationProvider { .build(); public OIDCAuthProvider(String aClientId, String aClientSecret, String issuerEndpointURL, - boolean pkceEnabled, String pkceMethod) throws AuthorizationSetupException { - this.clientSecret = aClientSecret; // nedded for state creation + boolean pkceEnabled, String pkceMethod, boolean jsfBlocked) throws AuthorizationSetupException { + this.clientSecret = aClientSecret; // needed for state creation this.clientAuth = new ClientSecretBasic(new ClientID(aClientId), new Secret(aClientSecret)); this.issuer = new Issuer(issuerEndpointURL); @@ -89,9 +90,11 @@ public OIDCAuthProvider(String aClientId, String aClientSecret, String issuerEnd this.pkceEnabled = pkceEnabled; this.pkceMethod = CodeChallengeMethod.parse(pkceMethod); + this.jsfBlocked = jsfBlocked; } - + public boolean isJsfBlocked() { return jsfBlocked; } + /** * Setup metadata from OIDC provider during creation of the provider representation * @return The OIDC provider metadata, if successfull diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java index 3f8c18d0567..7a2f6acc41c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java @@ -43,7 +43,8 @@ public AuthenticationProvider buildProvider( AuthenticationProviderRow aRow ) th factoryData.get("clientSecret"), factoryData.get("issuer"), Boolean.parseBoolean(factoryData.getOrDefault("pkceEnabled", "false")), - factoryData.getOrDefault("pkceMethod", "S256") + factoryData.getOrDefault("pkceMethod", "S256"), + Boolean.parseBoolean(factoryData.getOrDefault("jsfBlocked", "false")) ); oidc.setId(aRow.getId()); @@ -64,7 +65,8 @@ public static AuthenticationProvider buildFromSettings() throws AuthorizationSet JvmSettings.OIDC_CLIENT_SECRET.lookup(), JvmSettings.OIDC_AUTH_SERVER_URL.lookup(), JvmSettings.OIDC_PKCE_ENABLED.lookupOptional(Boolean.class).orElse(false), - JvmSettings.OIDC_PKCE_METHOD.lookupOptional().orElse("S256") + JvmSettings.OIDC_PKCE_METHOD.lookupOptional().orElse("S256"), + JvmSettings.OIDC_JSFBLOCKED.lookupOptional(Boolean.class).orElse(false) ); oidc.setId("oidc-mpconfig"); diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java index 87123801a3e..77c12ec3bd4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java @@ -252,6 +252,7 @@ public enum JvmSettings { OIDC_AUTH_SERVER_URL(SCOPE_OIDC, "auth-server-url"), OIDC_CLIENT_ID(SCOPE_OIDC, "client-id"), OIDC_CLIENT_SECRET(SCOPE_OIDC, "client-secret"), + OIDC_JSFBLOCKED(SCOPE_OIDC, "jsfBlocked"), SCOPE_OIDC_PKCE(SCOPE_OIDC, "pkce"), OIDC_PKCE_ENABLED(SCOPE_OIDC_PKCE, "enabled"), OIDC_PKCE_METHOD(SCOPE_OIDC_PKCE, "method"), diff --git a/src/main/webapp/loginpage.xhtml b/src/main/webapp/loginpage.xhtml index ffb2ce0f935..498f6e53ff3 100644 --- a/src/main/webapp/loginpage.xhtml +++ b/src/main/webapp/loginpage.xhtml @@ -23,7 +23,7 @@ - +

@@ -214,10 +214,10 @@
-
+

#{bundle['auth.providers.title']}

- + diff --git a/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java b/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java index b48c5507a54..45765b88ad5 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java @@ -18,6 +18,7 @@ import jakarta.json.Json; import jakarta.json.JsonArray; +import jakarta.json.JsonObject; import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.BeforeAll; @@ -40,6 +41,8 @@ public class AdminIT { private static final Logger logger = Logger.getLogger(AdminIT.class.getCanonicalName()); private final String testNonSuperuserApiToken = createTestNonSuperuserApiToken(); + static final String clientId = "test"; + static final String clientSecret = "94XHrfNRwXsjqTqApRrwWmhDLDHpIYV8"; @BeforeAll public static void setUp() { @@ -961,4 +964,40 @@ public void testSetSuperUserStatus(Boolean status) { toggleSuperuser.then().assertThat() .statusCode(OK.getStatusCode()); } + + // Testing creating an OIDC Provider not intended for use in JSF UI + @Test + public void testAddAuthProviders() { + Response createSuperuser = UtilIT.createRandomUser(); + String superuserUsername = UtilIT.getUsernameFromResponse(createSuperuser); + String superuserApiToken = UtilIT.getApiTokenFromResponse(createSuperuser); + Response toggleSuperuser = UtilIT.makeSuperUser(superuserUsername); + toggleSuperuser.then().assertThat() + .statusCode(OK.getStatusCode()); + + Response getAuthProviders = UtilIT.getAuthProviders(superuserApiToken); + getAuthProviders.prettyPrint(); + + String factoryData = String.format("type: oidc | issuer: http://keycloak.mydomain.com:8090/realms/test | clientId: %s | clientSecret: %s | jsfBlocked: true", clientId, clientSecret); + JsonObject jsonObject = Json.createObjectBuilder() + .add("id", "oidc1") + .add("factoryAlias", "oidc") + .add("title", "Open ID Connect SPA") + .add("subtitle", "SPA OIDC Provider") + .add("factoryData", factoryData) + .add("enabled", true) + .build(); + Response addAuthProviders = UtilIT.addAuthProviders(superuserApiToken, jsonObject); + addAuthProviders.prettyPrint(); + addAuthProviders.then().assertThat() + .statusCode(CREATED.getStatusCode()); + + getAuthProviders = UtilIT.getAuthProviders(superuserApiToken); + getAuthProviders.prettyPrint(); + getAuthProviders.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[1].id", equalTo("oidc1")) + .body("data[1].factoryData", containsString("jsfBlocked: true")); + + } } diff --git a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java index c11f66aa749..ebaa1dee2fd 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java @@ -1942,6 +1942,14 @@ static Response getAuthProviders(String apiToken) { .get("/api/admin/authenticationProviders"); return response; } + static Response addAuthProviders(String apiToken, JsonObject jsonObject) { + Response response = given() + .header(API_TOKEN_HTTP_HEADER, apiToken) + .body(jsonObject.toString()) + .contentType("application/json") + .post("/api/admin/authenticationProviders"); + return response; + } static Response migrateShibToBuiltin(Long userIdToConvert, String newEmailAddress, String apiToken) { Response response = given() From f6f3c4512108543c806b09aff8c9c6c659c8000b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 23 Oct 2025 19:32:18 +0000 Subject: [PATCH 048/578] Bump org.keycloak:keycloak-services in /conf/keycloak/builtin-users-spi Bumps [org.keycloak:keycloak-services](https://github.com/keycloak/keycloak) from 26.3.4 to 26.4.1. - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/26.3.4...26.4.1) --- updated-dependencies: - dependency-name: org.keycloak:keycloak-services dependency-version: 26.4.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- conf/keycloak/builtin-users-spi/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf/keycloak/builtin-users-spi/pom.xml b/conf/keycloak/builtin-users-spi/pom.xml index 2a730621f85..1d78eb763f9 100644 --- a/conf/keycloak/builtin-users-spi/pom.xml +++ b/conf/keycloak/builtin-users-spi/pom.xml @@ -100,7 +100,7 @@ - 26.3.4 + 26.4.1 17 3.2.0 0.4 From f74130c1122acc52a01fddbb0926cf742943633d Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 23 Oct 2025 16:31:56 -0400 Subject: [PATCH 049/578] addressing comments --- .../command/impl/MoveDatasetCommand.java | 24 +++++++---- .../edu/harvard/iq/dataverse/api/MoveIT.java | 41 +++++++++---------- 2 files changed, 36 insertions(+), 29 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java index 513e936dabf..a6609637fb9 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java @@ -42,10 +42,11 @@ public class MoveDatasetCommand extends AbstractVoidCommand { final Dataset moved; final Dataverse destination; final Boolean force; - private boolean allowSelfNotification = false; + final Boolean allowSelfNotification; + final Dataverse originalOwner; public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force) { - this( aRequest, moved, destination, force, Boolean.FALSE); + this( aRequest, moved, destination, force, null); } public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force, Boolean allowSelfNotification) { super( @@ -57,6 +58,7 @@ public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse de this.destination = destination; this.force= force; this.allowSelfNotification = allowSelfNotification; + this.originalOwner = moved.getOwner(); } @Override @@ -148,15 +150,20 @@ public void executeImpl(CommandContext ctxt) throws CommandException { } // OK, move - Dataverse originalOwner = moved.getOwner(); moved.setOwner(destination); ctxt.em().merge(moved); - sendNotification(moved, originalOwner, ctxt); boolean doNormalSolrDocCleanUp = true; ctxt.index().asyncIndexDataset(moved, doNormalSolrDocCleanUp); } + + @Override + public boolean onSuccess(CommandContext ctxt, Object r) { + sendNotification(moved, originalOwner, ctxt); + return true; + } + /** * Sends notifications to those able to publish the dataset upon the successful move of a dataset. *

@@ -181,14 +188,17 @@ protected void sendNotification(Dataset dataset, Dataverse originalOwner, Comman // 3. Get all users with publish permission on the dataset's original owner (dataverse) and notify them. Map recipients = ctxt.permissions().getDistinctUsersWithPermissionOn(Permission.PublishDataset, originalOwner); - // make sure the requestor is in the recipient list in case they don't match the permission + // make sure the requestor is in the recipient list in case they don't match the permission but only if allowSelfNotification is true if (requestor != null) { - recipients.put(requestor.getIdentifier(), requestor); + if (Boolean.TRUE.equals(allowSelfNotification)) { + recipients.put(requestor.getIdentifier(), requestor); + } else { + recipients.remove(requestor.getIdentifier()); + } } recipients.values() .stream() - .filter(recipient -> allowSelfNotification || !recipient.equals(requestor)) .forEach(recipient -> ctxt.notifications().sendNotification( recipient, Timestamp.from(Instant.now()), diff --git a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java index a547c44eea2..87cdd0a8f10 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java @@ -23,6 +23,7 @@ import static org.hamcrest.CoreMatchers.*; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -195,10 +196,6 @@ public void testMoveDatasetNotification() { .statusCode(CREATED.getStatusCode()); Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); - // clear existing notifications (so the DATASETMOVED notification will be the only one) - clearNotifications(user1ApiToken); - clearNotifications(user2ApiToken); - // User1(superuser) moves the dataset from dataverse2 to dataverse1 Response moveDataset = UtilIT.moveDataset(datasetId.toString(), dataverseAlias1, user1ApiToken); moveDataset.prettyPrint(); @@ -209,30 +206,30 @@ public void testMoveDatasetNotification() { // verify that a notification was sent to user1 Response getNotifications = UtilIT.getNotifications(user1ApiToken); getNotifications.prettyPrint(); - getNotifications.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.notifications[0].type", equalTo("DATASETMOVED")) - .body("data.notifications[0].displayAsRead", equalTo(false)) - .body("data.notifications[0].subjectText", containsString("has been moved")) - .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + verifyNotification(getNotifications, dataverseAlias1); + // verify that a notification was sent to user2 getNotifications = UtilIT.getNotifications(user2ApiToken); getNotifications.prettyPrint(); - getNotifications.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.notifications[0].type", equalTo("DATASETMOVED")) - .body("data.notifications[0].displayAsRead", equalTo(false)) - .body("data.notifications[0].subjectText", containsString("has been moved")) - .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + verifyNotification(getNotifications, dataverseAlias1); } - private void clearNotifications(String apiToken) { - Response getNotifications = UtilIT.getNotifications(apiToken); - List notifications = JsonPath.from(getNotifications.body().asString()).getList("data.notifications"); - for (Object obj : notifications) { - Object id = ((Map) obj).get("id"); - UtilIT.deleteNotification(Long.parseLong(id.toString()), apiToken).prettyPrint(); + private void verifyNotification(Response notificationListResponse, String dataverseAlias) { + notificationListResponse.then().assertThat() + .statusCode(OK.getStatusCode()); + boolean found = false; + List> notifications = notificationListResponse.body().jsonPath().getList("data.notifications"); + + for (Map notification : notifications) { + if ("DATASETMOVED".equalsIgnoreCase(notification.get("type"))) { + if (notification.get("messageText") != null && notification.get("messageText").contains(dataverseAlias)) { + found = true; + assertTrue(notification.get("subjectText") != null && notification.get("subjectText").contains("has been moved")); + assertTrue(notification.get("messageText") != null && notification.get("messageText").startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + } + } } + assertTrue(found); } @Test From 3f8dfd0ff9c0a176530403d57d2a5aab4eb415b5 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 23 Oct 2025 16:42:29 -0400 Subject: [PATCH 050/578] Revert "addressing comments" This reverts commit f74130c1122acc52a01fddbb0926cf742943633d. --- .../command/impl/MoveDatasetCommand.java | 24 ++++------- .../edu/harvard/iq/dataverse/api/MoveIT.java | 41 ++++++++++--------- 2 files changed, 29 insertions(+), 36 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java index a6609637fb9..513e936dabf 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java @@ -42,11 +42,10 @@ public class MoveDatasetCommand extends AbstractVoidCommand { final Dataset moved; final Dataverse destination; final Boolean force; - final Boolean allowSelfNotification; - final Dataverse originalOwner; + private boolean allowSelfNotification = false; public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force) { - this( aRequest, moved, destination, force, null); + this( aRequest, moved, destination, force, Boolean.FALSE); } public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force, Boolean allowSelfNotification) { super( @@ -58,7 +57,6 @@ public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse de this.destination = destination; this.force= force; this.allowSelfNotification = allowSelfNotification; - this.originalOwner = moved.getOwner(); } @Override @@ -150,20 +148,15 @@ public void executeImpl(CommandContext ctxt) throws CommandException { } // OK, move + Dataverse originalOwner = moved.getOwner(); moved.setOwner(destination); ctxt.em().merge(moved); + sendNotification(moved, originalOwner, ctxt); boolean doNormalSolrDocCleanUp = true; ctxt.index().asyncIndexDataset(moved, doNormalSolrDocCleanUp); } - - @Override - public boolean onSuccess(CommandContext ctxt, Object r) { - sendNotification(moved, originalOwner, ctxt); - return true; - } - /** * Sends notifications to those able to publish the dataset upon the successful move of a dataset. *

@@ -188,17 +181,14 @@ protected void sendNotification(Dataset dataset, Dataverse originalOwner, Comman // 3. Get all users with publish permission on the dataset's original owner (dataverse) and notify them. Map recipients = ctxt.permissions().getDistinctUsersWithPermissionOn(Permission.PublishDataset, originalOwner); - // make sure the requestor is in the recipient list in case they don't match the permission but only if allowSelfNotification is true + // make sure the requestor is in the recipient list in case they don't match the permission if (requestor != null) { - if (Boolean.TRUE.equals(allowSelfNotification)) { - recipients.put(requestor.getIdentifier(), requestor); - } else { - recipients.remove(requestor.getIdentifier()); - } + recipients.put(requestor.getIdentifier(), requestor); } recipients.values() .stream() + .filter(recipient -> allowSelfNotification || !recipient.equals(requestor)) .forEach(recipient -> ctxt.notifications().sendNotification( recipient, Timestamp.from(Instant.now()), diff --git a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java index 87cdd0a8f10..a547c44eea2 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java @@ -23,7 +23,6 @@ import static org.hamcrest.CoreMatchers.*; import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertTrue; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -196,6 +195,10 @@ public void testMoveDatasetNotification() { .statusCode(CREATED.getStatusCode()); Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + // clear existing notifications (so the DATASETMOVED notification will be the only one) + clearNotifications(user1ApiToken); + clearNotifications(user2ApiToken); + // User1(superuser) moves the dataset from dataverse2 to dataverse1 Response moveDataset = UtilIT.moveDataset(datasetId.toString(), dataverseAlias1, user1ApiToken); moveDataset.prettyPrint(); @@ -206,30 +209,30 @@ public void testMoveDatasetNotification() { // verify that a notification was sent to user1 Response getNotifications = UtilIT.getNotifications(user1ApiToken); getNotifications.prettyPrint(); - verifyNotification(getNotifications, dataverseAlias1); - + getNotifications.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.notifications[0].type", equalTo("DATASETMOVED")) + .body("data.notifications[0].displayAsRead", equalTo(false)) + .body("data.notifications[0].subjectText", containsString("has been moved")) + .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); // verify that a notification was sent to user2 getNotifications = UtilIT.getNotifications(user2ApiToken); getNotifications.prettyPrint(); - verifyNotification(getNotifications, dataverseAlias1); + getNotifications.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.notifications[0].type", equalTo("DATASETMOVED")) + .body("data.notifications[0].displayAsRead", equalTo(false)) + .body("data.notifications[0].subjectText", containsString("has been moved")) + .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); } - private void verifyNotification(Response notificationListResponse, String dataverseAlias) { - notificationListResponse.then().assertThat() - .statusCode(OK.getStatusCode()); - boolean found = false; - List> notifications = notificationListResponse.body().jsonPath().getList("data.notifications"); - - for (Map notification : notifications) { - if ("DATASETMOVED".equalsIgnoreCase(notification.get("type"))) { - if (notification.get("messageText") != null && notification.get("messageText").contains(dataverseAlias)) { - found = true; - assertTrue(notification.get("subjectText") != null && notification.get("subjectText").contains("has been moved")); - assertTrue(notification.get("messageText") != null && notification.get("messageText").startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); - } - } + private void clearNotifications(String apiToken) { + Response getNotifications = UtilIT.getNotifications(apiToken); + List notifications = JsonPath.from(getNotifications.body().asString()).getList("data.notifications"); + for (Object obj : notifications) { + Object id = ((Map) obj).get("id"); + UtilIT.deleteNotification(Long.parseLong(id.toString()), apiToken).prettyPrint(); } - assertTrue(found); } @Test From 75d0e90588fbf7df57a96f31f8c89c9f4f23463b Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 23 Oct 2025 16:43:38 -0400 Subject: [PATCH 051/578] Revert "Merge branch '11670-notification-of-moved-datasets' of https://github.com/IQSS/dataverse into 11670-notification-of-moved-datasets" This reverts commit 5a603fd4902c047975ee0cb5fcae69595a567cff, reversing changes made to f74130c1122acc52a01fddbb0926cf742943633d. --- .github/actions/setup-maven/action.yml | 2 +- .github/workflows/codeql.yml | 4 +- .github/workflows/container_app_pr.yml | 4 +- .github/workflows/container_maintenance.yml | 6 +- .github/workflows/deploy_beta_testing.yml | 2 +- .github/workflows/maven_cache_management.yml | 2 +- .github/workflows/maven_unit_test.yml | 6 +- .github/workflows/spi_release.yml | 10 +- conf/keycloak/builtin-users-spi/pom.xml | 2 +- conf/mdc/counter_weekly.sh | 92 ----- .../11387-modify-input-level-api.md | 3 - .../11695-change-api-get-storage-driver.md | 12 - .../11710-get-available-dataverses-api.md | 5 - .../11752-croissant-restricted.md | 13 - .../11766-new-io.gdcc.dataverse-spi.md | 2 - .../11777-MDC-citation-api-improvement.md | 7 - .../11783-Curation-Status-fixes.md | 1 - doc/release-notes/11800-qb-ra-fixes.md | 2 - .../11822-faster-permission-indexing.md | 1 - .../source/admin/dataverses-datasets.rst | 4 - .../source/admin/make-data-count.rst | 2 - doc/sphinx-guides/source/admin/monitoring.rst | 2 +- doc/sphinx-guides/source/api/changelog.rst | 6 - .../source/api/external-tools.rst | 4 +- doc/sphinx-guides/source/api/native-api.rst | 47 +-- .../source/container/app-image.rst | 2 +- .../source/container/running/demo.rst | 2 +- .../source/contributor/documentation.md | 84 ++-- .../source/developers/deployment.rst | 4 +- .../developers/making-library-releases.rst | 26 -- .../source/developers/making-releases.rst | 105 +---- .../source/developers/testing.rst | 16 +- .../source/installation/config.rst | 18 +- .../source/installation/prep.rst | 2 +- .../source/qa/testing-approach.md | 2 +- makefile | 6 +- modules/container-base/README.md | 2 +- modules/container-configbaker/README.md | 2 +- modules/dataverse-spi/pom.xml | 10 +- .../io/gdcc/spi/export/ExportDataContext.java | 61 --- .../io/gdcc/spi/export/ExportDataOption.java | 51 --- .../gdcc/spi/export/ExportDataProvider.java | 45 +-- .../java/io/gdcc/spi/export/Exporter.java | 1 + src/main/docker/README.md | 2 +- .../iq/dataverse/DatasetLinkingDataverse.java | 13 - .../dataverse/DatasetLinkingServiceBean.java | 44 +-- .../dataverse/DataverseLinkingDataverse.java | 11 - .../DataverseLinkingServiceBean.java | 37 +- .../iq/dataverse/DataverseServiceBean.java | 78 +--- .../edu/harvard/iq/dataverse/FilePage.java | 5 - .../iq/dataverse/PermissionServiceBean.java | 96 +---- .../iq/dataverse/api/AbstractApiBean.java | 10 +- .../edu/harvard/iq/dataverse/api/Admin.java | 12 +- .../harvard/iq/dataverse/api/Datasets.java | 4 +- .../dataverse/api/DataverseFeaturedItems.java | 2 +- .../harvard/iq/dataverse/api/Dataverses.java | 39 +- .../iq/dataverse/api/MakeDataCountApi.java | 172 ++------ .../providers/shib/ShibServiceBean.java | 4 +- .../impl/AbstractWriteDataverseCommand.java | 8 +- .../CuratePublishedDatasetVersionCommand.java | 25 +- .../impl/GetLinkingDataverseListCommand.java | 113 ------ .../iq/dataverse/export/ExportService.java | 1 - .../DatasetExternalCitationsServiceBean.java | 4 +- .../pidproviders/doi/XmlMetadataTemplate.java | 11 +- .../search/SolrIndexServiceBean.java | 211 +++++----- .../iq/dataverse/settings/JvmSettings.java | 4 - .../iq/dataverse/util/json/JsonPrinter.java | 50 +-- .../webapp/WEB-INF/glassfish-resources.xml | 10 - src/main/webapp/dataset.xhtml | 2 +- src/main/webapp/file.xhtml | 2 - .../guestbook-terms-popup-fragment.xhtml | 2 +- src/main/webapp/resources/css/structure.css | 3 - .../harvard/iq/dataverse/api/DatasetsIT.java | 118 +----- .../iq/dataverse/api/DataversesIT.java | 373 +----------------- .../harvard/iq/dataverse/api/S3AccessIT.java | 12 +- .../edu/harvard/iq/dataverse/api/UtilIT.java | 85 +--- .../impl/CreateDataverseCommandTest.java | 2 +- .../GetLinkingDataverseListCommandTest.java | 163 -------- 78 files changed, 320 insertions(+), 2088 deletions(-) delete mode 100644 conf/mdc/counter_weekly.sh delete mode 100644 doc/release-notes/11387-modify-input-level-api.md delete mode 100644 doc/release-notes/11695-change-api-get-storage-driver.md delete mode 100644 doc/release-notes/11710-get-available-dataverses-api.md delete mode 100644 doc/release-notes/11752-croissant-restricted.md delete mode 100644 doc/release-notes/11766-new-io.gdcc.dataverse-spi.md delete mode 100644 doc/release-notes/11777-MDC-citation-api-improvement.md delete mode 100644 doc/release-notes/11783-Curation-Status-fixes.md delete mode 100644 doc/release-notes/11800-qb-ra-fixes.md delete mode 100644 doc/release-notes/11822-faster-permission-indexing.md delete mode 100644 modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java delete mode 100644 modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java delete mode 100644 src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java delete mode 100644 src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java diff --git a/.github/actions/setup-maven/action.yml b/.github/actions/setup-maven/action.yml index 62ec7020b5b..e3e0de47329 100644 --- a/.github/actions/setup-maven/action.yml +++ b/.github/actions/setup-maven/action.yml @@ -23,7 +23,7 @@ runs: echo "JAVA_VERSION=$(grep '' ${GITHUB_WORKSPACE}/modules/dataverse-parent/pom.xml | cut -f2 -d'>' | cut -f1 -d'<')" | tee -a ${GITHUB_ENV} - name: Set up JDK ${{ env.JAVA_VERSION }} id: setup-java - uses: actions/setup-java@v5 + uses: actions/setup-java@v4 with: java-version: ${{ env.JAVA_VERSION }} distribution: 'temurin' diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 907452f4614..105469139ec 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -71,7 +71,7 @@ jobs: # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL - uses: github/codeql-action/init@v4 + uses: github/codeql-action/init@v3 with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} @@ -99,6 +99,6 @@ jobs: exit 1 - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v4 + uses: github/codeql-action/analyze@v3 with: category: "/language:${{matrix.language}}" diff --git a/.github/workflows/container_app_pr.yml b/.github/workflows/container_app_pr.yml index 898b46c2652..a5dddc755d1 100644 --- a/.github/workflows/container_app_pr.yml +++ b/.github/workflows/container_app_pr.yml @@ -23,7 +23,7 @@ jobs: - uses: actions/checkout@v5 with: ref: 'refs/pull/${{ github.event.client_payload.pull_request.number }}/merge' - - uses: actions/setup-java@v5 + - uses: actions/setup-java@v4 with: java-version: "17" distribution: 'adopt' @@ -86,7 +86,7 @@ jobs: :ship: [See on GHCR](https://github.com/orgs/gdcc/packages/container). Use by referencing with full name as printed above, mind the registry name. # Leave a note when things have gone sideways - - uses: peter-evans/create-or-update-comment@v5 + - uses: peter-evans/create-or-update-comment@v4 if: ${{ failure() }} with: issue-number: ${{ github.event.client_payload.pull_request.number }} diff --git a/.github/workflows/container_maintenance.yml b/.github/workflows/container_maintenance.yml index d863f838881..142363cbe1a 100644 --- a/.github/workflows/container_maintenance.yml +++ b/.github/workflows/container_maintenance.yml @@ -218,7 +218,7 @@ jobs: cat "./modules/container-base/README.md" - name: Push description to DockerHub for base image if: ${{ ! inputs.dry_run && ! inputs.damp_run && toJSON(needs.base-image.outputs.rebuilt_images) != '[]' }} - uses: peter-evans/dockerhub-description@v5 + uses: peter-evans/dockerhub-description@v4 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} @@ -243,7 +243,7 @@ jobs: cat "./src/main/docker/README.md" - name: Push description to DockerHub for application image if: ${{ ! inputs.dry_run && ! inputs.damp_run && toJSON(needs.application-image.outputs.rebuilt_images) != '[]' }} - uses: peter-evans/dockerhub-description@v5 + uses: peter-evans/dockerhub-description@v4 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} @@ -268,7 +268,7 @@ jobs: cat "./modules/container-configbaker/README.md" - name: Push description to DockerHub for config baker image if: ${{ ! inputs.dry_run && ! inputs.damp_run && toJSON(needs.configbaker-image.outputs.rebuilt_images) != '[]' }} - uses: peter-evans/dockerhub-description@v5 + uses: peter-evans/dockerhub-description@v4 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} diff --git a/.github/workflows/deploy_beta_testing.yml b/.github/workflows/deploy_beta_testing.yml index bf4a90d7fb4..0e060113ba0 100644 --- a/.github/workflows/deploy_beta_testing.yml +++ b/.github/workflows/deploy_beta_testing.yml @@ -16,7 +16,7 @@ jobs: steps: - uses: actions/checkout@v5 - - uses: actions/setup-java@v5 + - uses: actions/setup-java@v4 with: distribution: 'zulu' java-version: '17' diff --git a/.github/workflows/maven_cache_management.yml b/.github/workflows/maven_cache_management.yml index 6bfb567c90b..f266b804534 100644 --- a/.github/workflows/maven_cache_management.yml +++ b/.github/workflows/maven_cache_management.yml @@ -36,7 +36,7 @@ jobs: - name: Determine Java version from Parent POM run: echo "JAVA_VERSION=$(grep '' modules/dataverse-parent/pom.xml | cut -f2 -d'>' | cut -f1 -d'<')" >> ${GITHUB_ENV} - name: Set up JDK ${{ env.JAVA_VERSION }} - uses: actions/setup-java@v5 + uses: actions/setup-java@v4 with: java-version: ${{ env.JAVA_VERSION }} distribution: temurin diff --git a/.github/workflows/maven_unit_test.yml b/.github/workflows/maven_unit_test.yml index 5857b4489db..a416d5323f0 100644 --- a/.github/workflows/maven_unit_test.yml +++ b/.github/workflows/maven_unit_test.yml @@ -39,7 +39,7 @@ jobs: # Basic setup chores - uses: actions/checkout@v5 - name: Set up JDK ${{ matrix.jdk }} - uses: actions/setup-java@v5 + uses: actions/setup-java@v4 with: java-version: ${{ matrix.jdk }} distribution: temurin @@ -105,7 +105,7 @@ jobs: # Basic setup chores - uses: actions/checkout@v5 - name: Set up JDK ${{ matrix.jdk }} - uses: actions/setup-java@v5 + uses: actions/setup-java@v4 with: java-version: ${{ matrix.jdk }} distribution: temurin @@ -138,7 +138,7 @@ jobs: # TODO: As part of #10618 change to setup-maven custom action # Basic setup chores - uses: actions/checkout@v5 - - uses: actions/setup-java@v5 + - uses: actions/setup-java@v4 with: java-version: '17' distribution: temurin diff --git a/.github/workflows/spi_release.yml b/.github/workflows/spi_release.yml index 378e6ff9b67..9dc722c5992 100644 --- a/.github/workflows/spi_release.yml +++ b/.github/workflows/spi_release.yml @@ -38,11 +38,11 @@ jobs: if: github.event_name == 'pull_request' && needs.check-secrets.outputs.available == 'true' steps: - uses: actions/checkout@v5 - - uses: actions/setup-java@v5 + - uses: actions/setup-java@v4 with: java-version: '17' distribution: 'adopt' - server-id: central + server-id: ossrh server-username: MAVEN_USERNAME server-password: MAVEN_PASSWORD - uses: actions/cache@v4 @@ -64,7 +64,7 @@ jobs: if: github.event_name == 'push' && needs.check-secrets.outputs.available == 'true' steps: - uses: actions/checkout@v5 - - uses: actions/setup-java@v5 + - uses: actions/setup-java@v4 with: java-version: '17' distribution: 'adopt' @@ -76,11 +76,11 @@ jobs: # Running setup-java again overwrites the settings.xml - IT'S MANDATORY TO DO THIS SECOND SETUP!!! - name: Set up Maven Central Repository - uses: actions/setup-java@v5 + uses: actions/setup-java@v4 with: java-version: '17' distribution: 'adopt' - server-id: central + server-id: ossrh server-username: MAVEN_USERNAME server-password: MAVEN_PASSWORD gpg-private-key: ${{ secrets.DATAVERSEBOT_GPG_KEY }} diff --git a/conf/keycloak/builtin-users-spi/pom.xml b/conf/keycloak/builtin-users-spi/pom.xml index 2a730621f85..36cf6548d01 100644 --- a/conf/keycloak/builtin-users-spi/pom.xml +++ b/conf/keycloak/builtin-users-spi/pom.xml @@ -100,7 +100,7 @@ - 26.3.4 + 26.3.2 17 3.2.0 0.4 diff --git a/conf/mdc/counter_weekly.sh b/conf/mdc/counter_weekly.sh deleted file mode 100644 index 67cb5df2af2..00000000000 --- a/conf/mdc/counter_weekly.sh +++ /dev/null @@ -1,92 +0,0 @@ -#!/bin/sh -#counter_weekly.sh - -# This script iterates through all published Datasets in all Dataverses and calls the Make Data Count API to update their citations from DataCite -# Note: Requires curl and jq for parsing JSON responses form curl - -# A recursive method to process each Dataverse -processDV () { -echo "Processing Dataverse ID#: $1" - -#Call the Dataverse API to get the contents of the Dataverse (without credentials, this will only list published datasets and dataverses -DVCONTENTS=$(curl -s http://localhost:8080/api/dataverses/$1/contents) - -# Iterate over all datasets, pulling the value of their DOIs (as part of the persistentUrl) from the json returned -for subds in $(echo "${DVCONTENTS}" | jq -r '.data[] | select(.type == "dataset") | .persistentUrl'); do - -#The authority/identifier are preceded by a protocol/host, i.e. https://doi.org/ -DOI=`expr "$subds" : '.*:\/\/\doi\.org\/\(.*\)'` - -# Call the Dataverse API for this dataset and capture both the response and HTTP status code -HTTP_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "http://localhost:8080/api/admin/makeDataCount/:persistentId/updateCitationsForDataset?persistentId=doi:$DOI") - -# Extract the HTTP status code from the last line -HTTP_STATUS=$(echo "$HTTP_RESPONSE" | tail -n1) -# Extract the response body (everything except the last line) -RESPONSE_BODY=$(echo "$HTTP_RESPONSE" | sed '$d') - -# Check the HTTP status code and report accordingly -case $HTTP_STATUS in - 200) - # Successfully queued - # Extract status from the nested data object - STATUS=$(echo "$RESPONSE_BODY" | jq -r '.data.status') - - # Extract message from the nested data object - if echo "$RESPONSE_BODY" | jq -e '.data.message' > /dev/null 2>&1 && [ "$(echo "$RESPONSE_BODY" | jq -r '.data.message')" != "null" ]; then - MESSAGE=$(echo "$RESPONSE_BODY" | jq -r '.data.message') - echo "[SUCCESS] doi:$DOI - $STATUS: $MESSAGE" - else - # If message is missing or null, just show the status - echo "[SUCCESS] doi:$DOI - $STATUS: Citation update queued" - fi - ;; - 400) - # Bad request - if echo "$RESPONSE_BODY" | jq -e '.message' > /dev/null 2>&1; then - ERROR=$(echo "$RESPONSE_BODY" | jq -r '.message') - echo "[ERROR 400] doi:$DOI - Bad request: $ERROR" - else - echo "[ERROR 400] doi:$DOI - Bad request" - fi - ;; - 404) - # Not found - if echo "$RESPONSE_BODY" | jq -e '.message' > /dev/null 2>&1; then - ERROR=$(echo "$RESPONSE_BODY" | jq -r '.message') - echo "[ERROR 404] doi:$DOI - Not found: $ERROR" - else - echo "[ERROR 404] doi:$DOI - Not found" - fi - ;; - 503) - # Service unavailable (queue full) - if echo "$RESPONSE_BODY" | jq -e '.message' > /dev/null 2>&1; then - ERROR=$(echo "$RESPONSE_BODY" | jq -r '.message') - echo "[ERROR 503] doi:$DOI - Service unavailable: $ERROR" - elif echo "$RESPONSE_BODY" | jq -e '.data.message' > /dev/null 2>&1; then - ERROR=$(echo "$RESPONSE_BODY" | jq -r '.data.message') - echo "[ERROR 503] doi:$DOI - Service unavailable: $ERROR" - else - echo "[ERROR 503] doi:$DOI - Service unavailable: Queue is full" - fi - ;; - *) - # Other error - echo "[ERROR $HTTP_STATUS] doi:$DOI - Unexpected error" - echo "Response: $RESPONSE_BODY" - ;; -esac - -done - -# Now iterate over any child Dataverses and recursively process them -for subdv in $(echo "${DVCONTENTS}" | jq -r '.data[] | select(.type == "dataverse") | .id'); do -echo $subdv -processDV $subdv -done - -} - -# Call the function on the root dataverse to start processing -processDV 1 \ No newline at end of file diff --git a/doc/release-notes/11387-modify-input-level-api.md b/doc/release-notes/11387-modify-input-level-api.md deleted file mode 100644 index ca18eabb515..00000000000 --- a/doc/release-notes/11387-modify-input-level-api.md +++ /dev/null @@ -1,3 +0,0 @@ -### Update Collection Input Level API Changed - -- This endpoint will no longer delete the custom input levels previously modified for the given collection. In order to update a previously modified custom input level, it must be included in the JSON provided to the api. diff --git a/doc/release-notes/11695-change-api-get-storage-driver.md b/doc/release-notes/11695-change-api-get-storage-driver.md deleted file mode 100644 index aa993232f4d..00000000000 --- a/doc/release-notes/11695-change-api-get-storage-driver.md +++ /dev/null @@ -1,12 +0,0 @@ -## Get Dataset/Dataverse Storage Driver API - -### Changed Json response - breaking change! - -The API for getting the Storage Driver info has been changed/extended. -/api/datasets/{identifier}/storageDriver -/api/admin/dataverse/{dataverse-alias}/storageDriver -changed "message" to "name" and added "type" and "label" - -Also added query param for /api/admin/dataverse/{dataverse-alias}/storageDriver?getEffective=true to recurse the chain of parents to find the effective storageDriver - -See also [the guides](https://dataverse-guide--11664.org.readthedocs.build/en/11664/api/native-api.html#configure-a-dataset-to-store-all-new-files-in-a-specific-file-store), #11695, and #11664. diff --git a/doc/release-notes/11710-get-available-dataverses-api.md b/doc/release-notes/11710-get-available-dataverses-api.md deleted file mode 100644 index ac33c581848..00000000000 --- a/doc/release-notes/11710-get-available-dataverses-api.md +++ /dev/null @@ -1,5 +0,0 @@ -### New API endpoint for retrieving a list of Dataverse Collections to which a given Dataset or Dataverse Collection may be linked - --The end point also takes in a search term which currently must be part of the collections' names. --The user calling this API must have Link Dataset or Link Dataverse permission on the Dataverse Collections returned. --If the Collection has already been linked to the given Dataset or Collection, it will not be returned. diff --git a/doc/release-notes/11752-croissant-restricted.md b/doc/release-notes/11752-croissant-restricted.md deleted file mode 100644 index ced0ef4eaa7..00000000000 --- a/doc/release-notes/11752-croissant-restricted.md +++ /dev/null @@ -1,13 +0,0 @@ -- The optional Croissant exporter has been updated to 0.1.6 to prevent variable names, variable descriptions, and variable types from being exposed for restricted files. See https://github.com/gdcc/exporter-croissant/pull/20 and #11752. - -## Upgrade Instructions - -### Update Croissant exporter, if enabled, and reexport metadata - -If you have enabled the Croissant dataset metadata exporter, you should upgrade to version 0.1.6. - -- Stop Payara. -- Delete the old Croissant exporter jar file. It will be located in the directory defined by the `dataverse.spi.exporters.directory` setting. -- Download the updated Croissant jar from https://repo1.maven.org/maven2/io/gdcc/export/croissant/ and place it in the same directory. -- Restart Payara. -- Run reExportAll. diff --git a/doc/release-notes/11766-new-io.gdcc.dataverse-spi.md b/doc/release-notes/11766-new-io.gdcc.dataverse-spi.md deleted file mode 100644 index ef9e68f5719..00000000000 --- a/doc/release-notes/11766-new-io.gdcc.dataverse-spi.md +++ /dev/null @@ -1,2 +0,0 @@ -The ExportDataProvider framework in the dataverse-spi package has been extended, adding some extra options for developers of metadata exporter plugins. -See the [documentation](https://guides.dataverse.org/en/latest/developers/metadataexport.html#building-an-exporter) in the Metadata Export guide for details. \ No newline at end of file diff --git a/doc/release-notes/11777-MDC-citation-api-improvement.md b/doc/release-notes/11777-MDC-citation-api-improvement.md deleted file mode 100644 index 9441e9e0f44..00000000000 --- a/doc/release-notes/11777-MDC-citation-api-improvement.md +++ /dev/null @@ -1,7 +0,0 @@ -The /api/admin/makeDataCount/{id}/updateCitationsForDataset endpoint, which allows citations for a dataset to be retrieved from DataCite, is often called periodically for all datasets. However, allowing calls for many datasets to be processed in parallel can cause performance problems in Dataverse and/or cause calls to DataCite to fail due to rate limiting. The existing implementation was also inefficient w.r.t. memory use when used on datasets with many (>~1K) files. This release configures Dataverse to queue calls to this api, processes them serially, adds optional throttling to avoid hitting DataCite rate limits and improves memory use. - -New optional MPConfig setting: - -dataverse.api.mdc.min-delay-ms - number of milliseconds to wait between calls to DataCite. A value of ~100 should conservatively address DataCite's current 3000/5 minute limit. A value of 250 may be required for their test service. - -Backward compatibility: This api call is now asynchronous and will return an OK response when the call is queued or a 503 if the queue is full. \ No newline at end of file diff --git a/doc/release-notes/11783-Curation-Status-fixes.md b/doc/release-notes/11783-Curation-Status-fixes.md deleted file mode 100644 index 62b33165d0c..00000000000 --- a/doc/release-notes/11783-Curation-Status-fixes.md +++ /dev/null @@ -1 +0,0 @@ -In prior versions of Dataverse, publishing a dataset via the superuser only update-current-version option would not set the current curation status (if enabled/used) to none/empty and, in v6.7, would not maintain the curation status history. These issues are now resolved and the update current version option works the same as normal publication of a new version w.r.t. curation status. \ No newline at end of file diff --git a/doc/release-notes/11800-qb-ra-fixes.md b/doc/release-notes/11800-qb-ra-fixes.md deleted file mode 100644 index 5c7ef82f7f2..00000000000 --- a/doc/release-notes/11800-qb-ra-fixes.md +++ /dev/null @@ -1,2 +0,0 @@ -This release fixes problems with guestbook questions being displayed at download when files are selected from the dataset files table -when guestbook-at-request is enabled and not displaying when they should when access is requested from the file page. \ No newline at end of file diff --git a/doc/release-notes/11822-faster-permission-indexing.md b/doc/release-notes/11822-faster-permission-indexing.md deleted file mode 100644 index f716655232e..00000000000 --- a/doc/release-notes/11822-faster-permission-indexing.md +++ /dev/null @@ -1 +0,0 @@ -Permission reindexing, which occurs, e.g., after a user has been granted a role on a collection, has been made faster and less memory intensive in this release. \ No newline at end of file diff --git a/doc/sphinx-guides/source/admin/dataverses-datasets.rst b/doc/sphinx-guides/source/admin/dataverses-datasets.rst index 0fa5bcf69f1..a37819c90e1 100644 --- a/doc/sphinx-guides/source/admin/dataverses-datasets.rst +++ b/doc/sphinx-guides/source/admin/dataverses-datasets.rst @@ -60,10 +60,6 @@ The current driver can be seen using:: curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver -Or to recurse the chain of parents to find the effective storageDriver:: - - curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver?getEffective=true - (Note that for ``dataverse.files.store1.label=MyLabel``, ``store1`` will be returned.) and can be reset to the default store with:: diff --git a/doc/sphinx-guides/source/admin/make-data-count.rst b/doc/sphinx-guides/source/admin/make-data-count.rst index f8ffa7bb084..0103a6f9e38 100644 --- a/doc/sphinx-guides/source/admin/make-data-count.rst +++ b/doc/sphinx-guides/source/admin/make-data-count.rst @@ -166,8 +166,6 @@ The example :download:`counter_weekly.sh <../_static/util/counter_weekly.sh>` wi Citations will be retrieved for each published dataset and recorded in the your Dataverse installation's database. -Note that the :ref:`dataverse.api.mdc.min-delay-ms` setting can be used to avoid getting rate-limit errors from DataCite. - For how to get the citations out of your Dataverse installation, see "Retrieving Citations for a Dataset" under :ref:`Dataset Metrics ` in the :doc:`/api/native-api` section of the API Guide. Please note that while the Dataverse Software has a metadata field for "Related Dataset" this information is not currently sent as a citation to Crossref. diff --git a/doc/sphinx-guides/source/admin/monitoring.rst b/doc/sphinx-guides/source/admin/monitoring.rst index ef4e4f4f206..16bb18b7ad2 100644 --- a/doc/sphinx-guides/source/admin/monitoring.rst +++ b/doc/sphinx-guides/source/admin/monitoring.rst @@ -149,7 +149,7 @@ Tips: - Use **Enhanced Monitoring**. Enhanced Monitoring gathers its metrics from an agent on the instance. See `Enhanced Monitoring docs `_. - It's possible to view and act on **RDS Events** such as snapshots, parameter changes, etc. See `Working with Amazon RDS events `_ for details. - RDS monitoring is available via API and the ``aws`` command line tool. For example, see `Retrieving metrics with the Performance Insights API `_. -- To play with monitoring RDS using a server configured by `dataverse-ansible `_ set ``use_rds`` to true to skip some steps that aren't necessary when using RDS. See also the :doc:`/developers/deployment` section of the Developer Guide. +- To play with monitoring RDS using a server configured by `dataverse-ansible `_ set ``use_rds`` to true to skip some steps that aren't necessary when using RDS. See also the :doc:`/developers/deployment` section of the Developer Guide. MicroProfile Metrics endpoint ----------------------------- diff --git a/doc/sphinx-guides/source/api/changelog.rst b/doc/sphinx-guides/source/api/changelog.rst index d6523bfbdbc..5be6c78adce 100644 --- a/doc/sphinx-guides/source/api/changelog.rst +++ b/doc/sphinx-guides/source/api/changelog.rst @@ -7,18 +7,12 @@ This API changelog is experimental and we would love feedback on its usefulness. :local: :depth: 1 -v6.9 ----- -- The POST /api/admin/makeDataCount/{id}/updateCitationsForDataset processing is now asynchronous and the response no longer includes the number of citations. The response can be OK if the request is queued or 503 if the queue is full (default queue size is 1000). - v6.8 ---- - For POST /api/files/{id}/metadata passing an empty string ("description":"") or array ("categories":[]) will no longer be ignored. Empty fields will now clear out the values in the file's metadata. To ignore the fields simply do not include them in the JSON string. - For PUT /api/datasets/{id}/editMetadata the query parameter "sourceInternalVersionNumber" has been removed and replaced with "sourceLastUpdateTime" to verify that the data being edited hasn't been modified and isn't stale. - For GET /api/dataverses/$dataverse-alias/links the Json response has changed breaking the backward compatibility of the API. -- For GET /api/admin/dataverse/{dataverse-alias}/storageDriver and /api/datasets/{identifier}/storageDriver the driver name is no longer returned in data.message. This value is now returned in data.name. -- For PUT /api/dataverses/$dataverse-alias/inputLevels custom input levels that had been previously set will no longer be deleted. To delete input levels send an empty list (deletes all), then send the new/modified list. - For GET /api/externalTools and /api/externalTools/{id} the responses are now formatted as JSON (previously the toolParameters and allowedApiCalls were a JSON object and array (respectively) that were serialized as JSON strings) and any configured "requirements" are included. v6.7 diff --git a/doc/sphinx-guides/source/api/external-tools.rst b/doc/sphinx-guides/source/api/external-tools.rst index 389519318db..ae0e44b36aa 100644 --- a/doc/sphinx-guides/source/api/external-tools.rst +++ b/doc/sphinx-guides/source/api/external-tools.rst @@ -202,7 +202,7 @@ Testing Your External Tool As the author of an external tool, you are not expected to learn how to install and operate a Dataverse installation. There's a very good chance your tool can be added to a server Dataverse Community developers use for testing if you reach out on any of the channels listed under :ref:`getting-help-developers` in the Developer Guide. -By all means, if you'd like to install a Dataverse installation yourself, a number of developer-centric options are available. For example, there's a script to spin up a Dataverse installation on EC2 at https://github.com/gdcc/dataverse-ansible . The process for using curl to add your external tool to your Dataverse installation is documented under :ref:`managing-external-tools` in the Admin Guide. +By all means, if you'd like to install a Dataverse installation yourself, a number of developer-centric options are available. For example, there's a script to spin up a Dataverse installation on EC2 at https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible . The process for using curl to add your external tool to your Dataverse installation is documented under :ref:`managing-external-tools` in the Admin Guide. Spreading the Word About Your External Tool ------------------------------------------- @@ -219,7 +219,7 @@ If you've thought to yourself that there ought to be an app store for Dataverse Demoing Your External Tool ++++++++++++++++++++++++++ -https://demo.dataverse.org is the place to play around with the Dataverse Software and your tool can be included. Please email support@dataverse.org to start the conversation about adding your tool. Additionally, you are welcome to open an issue at https://github.com/gdcc/dataverse-ansible which already includes a number of the tools listed above. +https://demo.dataverse.org is the place to play around with the Dataverse Software and your tool can be included. Please email support@dataverse.org to start the conversation about adding your tool. Additionally, you are welcome to open an issue at https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible which already includes a number of the tools listed above. Announcing Your External Tool +++++++++++++++++++++++++++++ diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index a12dfa151c9..fa4b4611559 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -732,50 +732,6 @@ Note: you must have "Add Dataset" permission in the given collection to invoke t .. _featured-collections: -List Dataverse Collections to Which a Given Dataset or Dataverse Collection May Be Linked -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The user may provide a search term to limit the list of Dataverse Collections returned. The search term will be compared to the name of the Dataverse Collections. -The response is a JSON array of the ids, aliases, and names of the Dataverse collections to which a given Dataset or Dataverse Collection may be linked: - -For a given Dataverse Collection: - -.. code-block:: bash - - export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx - export SERVER_URL=https://demo.dataverse.org - export OBJECT_TYPE=dataverse - export ID=collectionAlias - export SEARCH_TERM=searchOn - - curl -H "X-Dataverse-key:$API_TOKEN" -X GET "$SERVER_URL/api/dataverses/$ID/$OBJECT_TYPE/linkingDataverses?searchTerm=$SEARCH_TERM" - -The fully expanded example above (without environment variables) looks like this: - -.. code-block:: bash - - curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/dataverses/collectionAlias/dataverse/linkingDataverses?searchTerm=searchOn" - -For a given Dataset: - -.. code-block:: bash - - export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx - export SERVER_URL=https://demo.dataverse.org - export OBJECT_TYPE=dataset - export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/J8SJZB - export SEARCH_TERM=searchOn - - curl -H "X-Dataverse-key:$API_TOKEN" -X GET "$SERVER_URL/api/dataverses/:persistentId/$OBJECT_TYPE/linkingDataverses?searchTerm=SEARCH_TERM&persistentId=$PERSISTENT_IDENTIFIER" - -The fully expanded example above (without environment variables) looks like this: - -.. code-block:: bash - - curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/dataverses/:persistentId/dataset/linkingDataverses?searchTerm=searchOn&persistentId=doi:10.5072/FK2/J8SJZB" - -You may also add an optional "alreadyLinked=true" parameter to return collections which are already linked to the given Dataset or Dataverse Collection. - List Featured Collections for a Dataverse Collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1150,8 +1106,7 @@ Update Collection Input Levels Updates the dataset field type input levels in a collection. -Please note that this endpoint does not change previously updated input levels of the collection page, so if you want to add new levels or modify existing ones, you will need to include them in the JSON request body. -In order to delete input levels you must call this API with an empty list to delete all of the input levels, then call this API with the new list of input levels. +Please note that this endpoint overwrites all the input levels of the collection page, so if you want to keep the existing ones, you will need to add them to the JSON request body. If one of the input levels corresponds to a dataset field type belonging to a metadata block that does not exist in the collection, the metadata block will be added to the collection. diff --git a/doc/sphinx-guides/source/container/app-image.rst b/doc/sphinx-guides/source/container/app-image.rst index 47055bd3786..afffeae1c0b 100644 --- a/doc/sphinx-guides/source/container/app-image.rst +++ b/doc/sphinx-guides/source/container/app-image.rst @@ -81,7 +81,7 @@ For now, stale images will be kept on Docker Hub indefinitely. | Example: :substitution-code:`|nextVersion|-noble` | Summary: Rolling tag, equivalent to ``unstable`` for current development cycle. Will roll over to the rolling production tag after a Dataverse release. - | Discussion: Perhaps you are eager to start testing features of an upcoming version (e.g. |nextVersion|) in a staging environment. You select the :substitution-code:`|nextVersion|-noble` tag (as opposed to ``unstable``) because you want to stay on |nextVersion| rather than switching to the version **after that** when a release is made (which would happen if you had selected the ``unstable`` tag). Also, when the next release comes out (|nextVersion| in this example), you would stay on the :substitution-code:`|nextVersion|-noble` tag, which is the same tag that someone would use who wants the final release of |nextVersion|. (See "Rolling Production", above.) + | Discussion: Perhaps you are eager to starting testing features of an upcoming version (e.g. |nextVersion|) in a staging environment. You select the :substitution-code:`|nextVersion|-noble` tag (as opposed to ``unstable``) because you want to stay on |nextVersion| rather switching to the version **after that** when a release is made (which would happen if you had selected the ``unstable`` tag). Also, when the next release comes out (|nextVersion| in this example), you would stay on the :substitution-code:`|nextVersion|-noble` tag, which is the same tag that someone would use who wants the final release of |nextVersion|. (See "Rolling Production", above.) **NOTE**: In these tags for development usage, the version number will always be 1 minor version ahead of existing Dataverse releases. Example: Assume Dataverse ``6.x`` is released, ``6.(x+1)`` is underway. diff --git a/doc/sphinx-guides/source/container/running/demo.rst b/doc/sphinx-guides/source/container/running/demo.rst index 32de0ea48bf..d4afee8a18a 100644 --- a/doc/sphinx-guides/source/container/running/demo.rst +++ b/doc/sphinx-guides/source/container/running/demo.rst @@ -261,7 +261,7 @@ You should be able to see the new fields from the metadata block you added in th ``curl http://localhost:8983/solr/collection1/schema/fields`` -At this point you can proceed with testing the metadata block in the Dataverse UI. First you'll need to enable it for a collection (see :ref:`general-information` in the User Guide section about collections). Afterwards, create a new dataset, save it, and then edit the metadata for that dataset. Your metadata block should appear. +At this point you can proceed with testing the metadata block in the Dataverse UI. First you'll need to enable it for a collection (see :ref:`general-information` in the User Guide section about collection). Afterwards, create a new dataset, save it, and then edit the metadata for that dataset. Your metadata block should appear. Next Steps ---------- diff --git a/doc/sphinx-guides/source/contributor/documentation.md b/doc/sphinx-guides/source/contributor/documentation.md index 614dba1d6a0..2a8d6794921 100644 --- a/doc/sphinx-guides/source/contributor/documentation.md +++ b/doc/sphinx-guides/source/contributor/documentation.md @@ -50,78 +50,76 @@ If you would like to read more about the Dataverse's use of GitHub, please see t ## Building the Guides with Sphinx -While the "quick fix" technique shown above should work fine for minor changes, in many cases, you're going to want to preview changes locally before committing them. +While the "quick fix" technique shown above should work fine for minor changes, especially for larger changes, we recommend installing Sphinx on your computer or using a Sphinx Docker container to build the guides locally so you can get an accurate preview of your changes. -Before we worry about pushing changes to the code, let's make sure we can build the guides. +In case you decide to use a Sphinx Docker container to build the guides, you can skip the next two installation sections, but you will need to have Docker installed. -Go to and click "Code" and then follow the instructions to clone the code locally. +### Installing Sphinx -### Docker +First, make a fork of and clone your fork locally. Then change to the ``doc/sphinx-guides`` directory. -Install [Docker Desktop](https://www.docker.com/products/docker-desktop/). +``cd doc/sphinx-guides`` -From a terminal, switch to the "dataverse" directory you just cloned. This is the root of the git repo. - -`cd dataverse` +Create a Python virtual environment, activate it, then install dependencies: -Then try running this command: +``python3 -m venv venv`` -`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx:7.2.6 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make html"` +``source venv/bin/activate`` -If all goes well, you should be able to open `doc/sphinx-guides/build/html/index.html` to see the guides you just built. +``pip install -r requirements.txt`` -#### Docker with a Makefile +### Installing GraphViz -Once you've confirmed you have Docker working, if you have [make](https://en.wikipedia.org/wiki/Make_(software)) installed, you can try the following commands: +In some parts of the documentation, graphs are rendered as images using the Sphinx GraphViz extension. -`make docs-html` +Building the guides requires the ``dot`` executable from GraphViz. -`make docs-pdf` +This requires having [GraphViz](https://graphviz.org) installed and either having ``dot`` on the path or +[adding options to the `make` call](https://groups.google.com/forum/#!topic/sphinx-users/yXgNey_0M3I). -`make docs-epub` +On a Mac we recommend installing GraphViz through [Homebrew](). Once you have Homebrew installed and configured to work with your shell, you can type `brew install graphviz`. -`make docs-all` +### Editing and Building the Guides -### Sphinx Installed Locally +To edit the existing documentation: -First, run `python --version` or `python3 --version` to determine the version of Python you have. If you don't have Python 3.10 or higher, you must upgrade. +- Create a branch (see {ref}`how-to-make-a-pull-request`). +- In ``doc/sphinx-guides/source`` you will find the .rst files that correspond to https://guides.dataverse.org. +- Using your preferred text editor, open and edit the necessary files, or create new ones. -Next, change to the `doc/sphinx-guides` directory. +Once you are done, you can preview the changes by building the guides locally. As explained, you can build the guides with Sphinx locally installed, or with a Docker container. -`cd doc/sphinx-guides` +#### Building the Guides with Sphinx Installed Locally -Create a Python virtual environment, activate it, then install dependencies: +Open a terminal, change directories to `doc/sphinx-guides`, activate (or reactivate) your Python virtual environment, and build the guides. -`python3 -m venv venv` +`cd doc/sphinx-guides` `source venv/bin/activate` -`pip install -r requirements.txt` - -Next, install [GraphViz](https://graphviz.org) because building the guides requires having the `dot` executable from GraphViz either on the path or passed [as an argument](https://groups.google.com/g/sphinx-users/c/yXgNey_0M3I/m/3T2NipFlBgAJ). +`make clean` -On a Mac we recommend installing GraphViz through [Homebrew](). Once you have Homebrew installed and configured to work with your shell, you can type `brew install graphviz`. +`make html` -Finally, you can try building the guides with the following command. +#### Building the Guides with a Sphinx Docker Container and a Makefile -`make html` +We have added a Makefile to simplify the process of building the guides using a Docker container, you can use the following commands from the repository root: -If all goes well, you should be able to open `doc/sphinx-guides/build/html/index.html` to see the guides you just built. +- `make docs-html` +- `make docs-pdf` +- `make docs-epub` +- `make docs-all` -## Editing, Building, and Previewing the Guides +#### Building the Guides with a Sphinx Docker Container and CLI -To edit the existing documentation: +If you want to build the guides using a Docker container, execute the following command in the repository root: -- Create a branch (see {ref}`how-to-make-a-pull-request`). -- In `doc/sphinx-guides/source` you will find the .rst or .md files that correspond to https://guides.dataverse.org. -- Using your preferred text editor, open and edit the necessary files, or create new ones. +`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx:7.2.6 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make html"` -Once you are done, you can preview the changes by building the guides using one of the options above. +#### Previewing the Guides After Sphinx is done processing the files you should notice that the `html` folder in `doc/sphinx-guides/build` directory has been updated. You can click on the files in the `html` folder to preview the changes. -## Making a Pull Request - Now you can make a commit with the changes to your own fork in GitHub and submit a pull request. See {ref}`how-to-make-a-pull-request`. ## Writing Guidelines @@ -155,22 +153,16 @@ If the page is written in Markdown (.md), use this form: ### Links -Getting links right can be tricky. +Getting links right with .rst files can be tricky. #### Custom Titles -In .rst files you can use a custom title when linking to a document like this: +You can use a custom title when linking to a document like this: :doc:`Custom title ` See also -In .md files, the same pattern can be used. Here's an example of using a custom title with a ref: - - {ref}`Log in ` - -See also - ### Images A good documentation is just like a website enhanced and upgraded by adding high quality and self-explanatory images. Often images depict a lot of written text in a simple manner. Within our Sphinx docs, you can add them in two ways: a) add a PNG image directly and include or b) use inline description languages like GraphViz (current only option). diff --git a/doc/sphinx-guides/source/developers/deployment.rst b/doc/sphinx-guides/source/developers/deployment.rst index ec9929136b7..46cf95dae54 100755 --- a/doc/sphinx-guides/source/developers/deployment.rst +++ b/doc/sphinx-guides/source/developers/deployment.rst @@ -78,7 +78,7 @@ Amazon offers instructions on using an IAM role to grant permissions to applicat Configure Ansible File (Optional) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In order to configure Dataverse installation settings such as the password of the dataverseAdmin user, download https://raw.githubusercontent.com/gdcc/dataverse-ansible/develop/defaults/main.yml and edit the file to your liking. +In order to configure Dataverse installation settings such as the password of the dataverseAdmin user, download https://raw.githubusercontent.com/GlobalDataverseCommunityConsortium/dataverse-ansible/master/defaults/main.yml and edit the file to your liking. You can skip this step if you're fine with the values in the "main.yml" file in the link above. @@ -89,7 +89,7 @@ Once you have done the configuration above, you are ready to try running the "ec Download `ec2-create-instance.sh`_ and put it somewhere reasonable. For the purpose of these instructions we'll assume it's in the "Downloads" directory in your home directory. -.. _ec2-create-instance.sh: https://raw.githubusercontent.com/gdcc/dataverse-ansible/develop/ec2/ec2-create-instance.sh +.. _ec2-create-instance.sh: https://raw.githubusercontent.com/GlobalDataverseCommunityConsortium/dataverse-ansible/master/ec2/ec2-create-instance.sh To run the script, you can make it executable (``chmod 755 ec2-create-instance.sh``) or run it with bash, like this with ``-h`` as an argument to print the help: diff --git a/doc/sphinx-guides/source/developers/making-library-releases.rst b/doc/sphinx-guides/source/developers/making-library-releases.rst index 0daa7fb89db..be867f9196a 100755 --- a/doc/sphinx-guides/source/developers/making-library-releases.rst +++ b/doc/sphinx-guides/source/developers/making-library-releases.rst @@ -36,32 +36,6 @@ Releasing a Snapshot Version to Maven Central That is to say, to make a snapshot release, you only need to get one or more commits into the default branch. -It's possible, of course, to make snapshot releases outside of GitHub Actions, from environments such as your laptop. Generally, you'll want to look at the GitHub Action and try to do the equivalent. You'll need a file set up locally at ``~/.m2/settings.xml`` with the following (contact a core developer for the redacted bits): - -.. code-block:: bash - - - - - central - REDACTED - REDACTED - - - - -Then, study the GitHub Action and perform similar commands from your local environment. For example, as of this writing, for the dataverse-spi project, you can run the following commands, substituting the suffix you need: - -``mvn -f modules/dataverse-spi -Dproject.version.suffix="2.1.0-PR11767-SNAPSHOT" verify`` - -``mvn -f modules/dataverse-spi -Dproject.version.suffix="2.1.0-PR11767-SNAPSHOT" deploy`` - -This will upload the snapshot here, for example: https://central.sonatype.com/repository/maven-snapshots/io/gdcc/dataverse-spi/2.1.02.1.0-PR11767-SNAPSHOT/dataverse-spi-2.1.02.1.0-PR11767-20250827.182026-1.jar - -Before OSSRH was retired, you could browse through snapshot jars you published at https://s01.oss.sonatype.org/content/repositories/snapshots/io/gdcc/dataverse-spi/2.0.0-PR9685-SNAPSHOT/, for example. Now, even though you may see the URL of the jar as shown above during the "deploy" step, if you try to browse the various snapshot jars at https://central.sonatype.com/repository/maven-snapshots/io/gdcc/dataverse-spi/2.1.02.1.0-PR11767-SNAPSHOT/ you'll see "This maven2 hosted repository is not directly browseable at this URL. Please use the browse or HTML index views to inspect the contents of this repository." Sadly, the "browse" and "HTML index" links don't work, as noted in a `question `_ on the Sonatype Community forum. Below is a suggestion for confirming that the jar was uploaded properly, which is to use Maven to copy the jar to your local directory. You could then compare checksums. - -``mvn dependency:copy -DrepoUrl=https://central.sonatype.com/repository/maven-snapshots/ -Dartifact=io.gdcc:dataverse-spi:2.1.02.1.0-PR11767-SNAPSHOT -DoutputDirectory=.`` - Releasing a Release (Non-Snapshot) Version to Maven Central ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index fbbc2e5d3ae..028b80e2892 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -8,13 +8,9 @@ Making Releases Introduction ------------ -.. note:: This document is about releasing the main Dataverse app (https://github.com/IQSS/dataverse). See :doc:`making-library-releases` for how to release our various libraries. Other projects have their own release documentation. +This document is about releasing the main Dataverse app (https://github.com/IQSS/dataverse). See :doc:`making-library-releases` for how to release our various libraries. Other projects have their own release documentation. -.. note:: Below you'll see branches like "develop" and "master" mentioned. For more on our branching strategy, see :doc:`version-control`. - -Dataverse releases are time-based as opposed to being feature-based. That is, we announce an approximate release date in advance (e.g. for `6.8 `_) and try to hit that deadline. If features we're working on aren't ready yet, the train will leave the station without them. We release quarterly. - -We also announce "last call" dates for both community pull requests and those made by core developers. If you are part of the community and have made a pull request, you have until this date to ask the team to add the upcoming milestone to your pull request. The same goes for core developers. This is not a guarantee that these pull requests will be reviewed, tested, QA'ed and merged before :ref:`code freeze `, but we'll try. +Below you'll see branches like "develop" and "master" mentioned. For more on our branching strategy, see :doc:`version-control`. Regular or Hotfix? ------------------ @@ -34,9 +30,7 @@ Early on, make sure it's clear what type of release this is. The steps below des Ensure Issues Have Been Created ------------------------------- -We have a "create release issues" script at https://github.com/IQSS/dv-project-metrics that should be run a week or so before code freeze. - -For each issue that is created by the script there is likely a corresponding step in this document that has "dedicated" label on it like this: +Some of the steps in this document are well-served by having their own dedicated GitHub issue. You'll see a label like this on them: |dedicated| @@ -47,39 +41,35 @@ There are a variety of reasons why a step might deserve its own dedicated issue: Steps don't get their own dedicated issue if it would be confusing to have multiple people involved. Too many cooks in the kitchen, as they say. Also, some steps are so small the overhead of an issue isn't worth it. +Before the release even begins you can coordinate with the project manager about the creation of these issues. + .. |dedicated| raw:: html Dedicated Issue   -.. _declare-code-freeze: - Declare a Code Freeze --------------------- -When we declare a code freeze, we mean: - -- No additional features will be merged until the freeze is lifted. -- Bug fixes will only be merged if they relate to the upcoming release in some way, such as fixes for regressions or performance problems in that release. -- Pull requests that directly affect the release, such as bumping the version, will be merged, of course. +The following steps are made more difficult if code is changing in the "develop" branch. Declare a code freeze until the release is out. Do not allow pull requests to be merged. -The benefits of the code freeze are: +For a hotfix, a code freeze (no merging) is necessary not because we want code to stop changing in the branch being hotfix released, but because bumping the version used in Jenkins/Ansible means that API tests will fail in pull requests until the version is bumped in those pull requests. -- The team can focus on getting the release out together. -- Regression and performance testing can happen on code that isn't changing. -- The release notes can be written without having to worry about new features (and their release note snippets) being merged in. +Conduct Performance Testing +--------------------------- -In short, the steps described below become easier under a code freeze. +|dedicated| -Note: for a hotfix, a code freeze is necessary not because we want code to stop changing in the branch being hotfix released, but because bumping the version used in Jenkins/Ansible means that API tests will fail in pull requests until the version is bumped in those pull requests. Basically, we want to get the hotfix merged quickly so we can propagate the version bump into all open pull requests so that API tests can start passing again in those pull requests. +See :doc:`/qa/performance-tests` for details. -Push Back Milestones on Pull Requests That Missed the Train ------------------------------------------------------------ +Conduct Regression Testing +--------------------------- -As of this writing, we optimistically add milestones to issues and pull requests, hoping that the work will be complete before code freeze. Inevitably, we're a bit too optimistic. +|dedicated| -Hopefully, as the release approached, the team has already decided which pull requests (that aren't related to the release) won't make the cut. If not, go ahead and bump them to the next release. +See :doc:`/qa/testing-approach` for details. +Refer to the provided regression checklist for the list of items to verify during the testing process: `Regression Checklist `_. .. _write-release-notes: @@ -95,7 +85,7 @@ The task at or near release time is to collect these snippets into a single file - Find the issue in GitHub that tracks the work of creating release notes for the upcoming release. - Create a branch, add a .md file for the release (ex. 5.10.1 Release Notes) in ``/doc/release-notes`` and write the release notes, making sure to pull content from the release note snippets mentioned above. Snippets may not include any issue number or pull request number in the text so be sure to copy the number from the filename of the snippet into the final release note. - Delete (``git rm``) the release note snippets as the content is added to the main release notes file. -- Include instructions describing the steps required to upgrade the application from the previous version. These must be customized for release numbers and special circumstances such as changes to metadata blocks and infrastructure. These instructions are required for the next steps (deploying to various environments) so try to prioritize them over finding just the right words in release highlights (which you can do later). +- Include instructions describing the steps required to upgrade the application from the previous version. These must be customized for release numbers and special circumstances such as changes to metadata blocks and infrastructure. - Make a pull request. Here's an example: https://github.com/IQSS/dataverse/pull/11613 - Note that we won't merge the release notes until after we have confirmed that the upgrade instructions are valid by performing a couple upgrades. @@ -120,58 +110,12 @@ ssh into the dataverse-internal server and download the release candidate war fi Go to /doc/release-notes, open the release-notes.md file for the release we're working on, and perform all the steps under "Upgrade Instructions". Note that for regular releases, we haven't bumped the version yet so you won't be able to follow the steps exactly. (For hotfix releases, the version will be bumped already.) -Deploy Release Candidate to QA ------------------------------- - -|dedicated| - -Deploy the same war file to https://qa.dataverse.org using the same upgrade instructions as above. - -Solicit Feedback from Curation Team ------------------------------------ - -Ask the curation team to test on https://qa.dataverse.org and give them five days to provide feedback. - - -Conduct Performance Testing ---------------------------- - -|dedicated| - -See :doc:`/qa/performance-tests` for details. - -Conduct Regression Testing ---------------------------- - -|dedicated| - -Regression testing should be conducted on production data. -See :doc:`/qa/testing-approach` for details. -Refer to the provided regression checklist for the list of items to verify during the testing process: `Regression Checklist `_. - -Build the Guides for the Release Candidate ------------------------------------------- - -Go to https://jenkins.dataverse.org/job/guides.dataverse.org/ and make the following adjustments to the config: - -- Repository URL: ``https://github.com/IQSS/dataverse.git`` -- Branch Specifier (blank for 'any'): ``*/develop`` -- ``VERSION`` (under "Build Steps"): use the next release version but add "-rc.1" to the end. Don't prepend a "v". Use ``6.8-rc.1`` (for example) - -Click "Save" then "Build Now". - -Make sure the guides directory appears in the expected location such as https://guides.dataverse.org/en/6.8-rc.1/ - -When previewing the HTML version of docs from pull requests, we don't usually use this Jenkins job, relying instead on automated ReadTheDocs builds. The reason for doing this step now while we wait for feedback from the Curation Team is that it's an excellent time to fix the Jenkins job, if necessary, to accommodate any changes needed to continue to build the docs. For example, Sphinx might need to be updated or a dependency might need to be installed. Such changes should be listed in the release notes for documentation writers. - Deploy Release Candidate to Demo -------------------------------- |dedicated| -Time has passed. The curation team has given feedback. We've finished regression and performance testing. Fixes may have been merged into the "develop" branch. We're ready to actually make the release now, which includes deploying a release candidate to the demo server. - -Build a new war file, if necessary, and deploy it to https://demo.dataverse.org using the upgrade instructions in the release notes. +Deploy the same war file to https://demo.dataverse.org using the same upgrade instructions as above. Merge Release Notes (Once Ready) -------------------------------- @@ -227,25 +171,16 @@ Merge "develop" into "master" (non-hotfix only) If this is a regular (non-hotfix) release, create a pull request to merge the "develop" branch into the "master" branch using this "compare" link: https://github.com/IQSS/dataverse/compare/master...develop -Allow time for important tests (compile, unit tests, etc.) to pass. Don't worry about style tests failing such as for shell scripts. It's ok to skip code review. - -When merging the pull request, be sure to choose "create a merge commit" and not "squash and merge" or "rebase and merge". We suspect that choosing squash or rebase may have led to `lots of merge conflicts `_ when we tried to perform this "merge develop to master" step, forcing us to `re-do `_ the previous release before we could proceed with the current release. +Once important tests have passed (compile, unit tests, etc.), merge the pull request (skipping code review is ok). Don't worry about style tests failing such as for shell scripts. If this is a hotfix release, skip this whole "merge develop to master" step (the "develop" branch is not involved until later). -Confirm "master" Mergeability ------------------------------ - -Hopefully, the previous step went ok. As a sanity check, use the "compare" link at https://github.com/IQSS/dataverse/compare/master...develop again to look for merge conflicts without making a pull request. - -If the GitHub UI tells you there would be merge conflicts, something has gone horribly wrong (again) with the "merge develop to master" step. Stop and ask for help. - Add Milestone to Pull Requests and Issues ----------------------------------------- Often someone is making sure that the proper milestone (e.g. 5.10.1) is being applied to pull requests and issues, but sometimes this falls between the cracks. -Check for merged pull requests that have no milestone by going to https://github.com/IQSS/dataverse/pulls and entering `is:pr is:merged no:milestone `_ as a query. If you find any, first check if those pull requests are against open pull requests. If so, do nothing. Otherwise, add the milestone to the pull request and any issues it closes. This includes the "merge develop into master" pull request above. +Check for merged pull requests that have no milestone by going to https://github.com/IQSS/dataverse/pulls and entering `is:pr is:merged no:milestone `_ as a query. If you find any, add the milestone to the pull request and any issues it closes. This includes the "merge develop into master" pull request above. (Optional) Test Docker Images ----------------------------- diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index f84a7cf1ac7..1690864d453 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -128,7 +128,7 @@ You might find studying the following test classes helpful in writing tests for - DeletePrivateUrlCommandTest.java - GetPrivateUrlCommandTest.java -In addition, there is a writeup on "The Testable Command" at https://github.com/IQSS/dataverse/blob/master/doc/theTestableCommand/TheTestableCommand.md . +In addition, there is a writeup on "The Testable Command" at https://github.com/IQSS/dataverse/blob/develop/doc/theTestableCommand/TheTestableCommand.md . Running Non-Essential (Excluded) Unit Tests ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -169,12 +169,12 @@ different people. For our purposes, an integration test can have two flavors: Running the Full API Test Suite Using EC2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**Prerequisite:** To run the API test suite in an EC2 instance you should first follow the steps in the :doc:`deployment` section to get set up with the AWS binary to launch EC2 instances. If you're here because you just want to spin up a branch, you'll still want to follow the AWS deployment setup steps, but may find the `ec2-create README.md `_ Quick Start section helpful. +**Prerequisite:** To run the API test suite in an EC2 instance you should first follow the steps in the :doc:`deployment` section to get set up with the AWS binary to launch EC2 instances. If you're here because you just want to spin up a branch, you'll still want to follow the AWS deployment setup steps, but may find the `ec2-create README.md `_ Quick Start section helpful. -You may always retrieve a current copy of the ec2-create-instance.sh script and accompanying group_var.yml file from the `dataverse-ansible repo `_. Since we want to run the test suite, let's grab the group_vars used by Jenkins: +You may always retrieve a current copy of the ec2-create-instance.sh script and accompanying group_var.yml file from the `dataverse-ansible repo `_. Since we want to run the test suite, let's grab the group_vars used by Jenkins: -- `ec2-create-instance.sh `_ -- `jenkins.yml `_ +- `ec2-create-instance.sh `_ +- `jenkins.yml `_ Edit ``jenkins.yml`` to set the desired GitHub repo and branch, and to adjust any other options to meet your needs: @@ -184,7 +184,7 @@ Edit ``jenkins.yml`` to set the desired GitHub repo and branch, and to adjust an - ``dataverse.unittests.enabled: true`` - ``dataverse.sampledata.enabled: true`` -If you wish, you may pass the script a ``-l`` flag with a local relative path in which the script will `copy various logs `_ at the end of the test suite for your review. +If you wish, you may pass the script a ``-l`` flag with a local relative path in which the script will `copy various logs `_ at the end of the test suite for your review. Finally, run the script: @@ -526,7 +526,7 @@ Browser-Based Testing Installation Testing ~~~~~~~~~~~~~~~~~~~~ -- Work with @donsizemore to automate testing of https://github.com/gdcc/dataverse-ansible +- Work with @donsizemore to automate testing of https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible Future Work on Load/Performance Testing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -538,4 +538,4 @@ Future Work on Load/Performance Testing Future Work on Accessibility Testing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Using https://github.com/gdcc/dataverse-ansible and hooks available from accessibility testing tools, automate the running of accessibility tools on PRs so that developers will receive quicker feedback on proposed code changes that reduce the accessibility of the application. +- Using https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible and hooks available from accessibility testing tools, automate the running of accessibility tools on PRs so that developers will receive quicker feedback on proposed code changes that reduce the accessibility of the application. diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 171320919d9..14bf33c9482 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -3189,7 +3189,7 @@ dataverse.person-or-org.org-phrase-array Please note that this setting is experimental. The Schema.org metadata and OpenAIRE exports and the Schema.org metadata included in DatasetPages try to infer whether each entry in the various fields (e.g. Author, Contributor) is a Person or Organization. -If you have examples where an organization name is being inferred to belong to a person, you can use this setting to force it to be recognized as an organization. +If you have examples where an orgization name is being inferred to belong to a person, you can use this setting to force it to be recognized as an organization. The value is expected to be a comma-separated list of strings. Any name that contains one of the strings is assumed to be an organization. For example, "Project" is a word that is not otherwise associated with being an organization. Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_PERSON_OR_ORG_ORG_PHRASE_ARRAY``. @@ -3731,22 +3731,6 @@ Example: Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_CORS_HEADERS_EXPOSE``. - -.. _dataverse.api.mdc.min-delay-ms: - -dataverse.api.mdc.min-delay-ms -++++++++++++++++++++++++++++++ - -Minimum delay in milliseconds between Make Data Count (MDC) API requests from the /api/admin/makeDataCount/{id}/updateCitationsForDataset api. -This setting helps prevent overloading the MDC service by enforcing a minimum time interval between consecutive requests. -If a request arrives before this interval has elapsed since the previous request, it will be rate-limited. - -Default: ``0`` (no delay enforced) - -Example: ``dataverse.api.mdc.min-delay-ms=100`` (enforces a minimum 100ms delay between MDC API requests) - -Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_API_MDC_MIN_DELAY_MS``. - .. _feature-flags: Feature Flags diff --git a/doc/sphinx-guides/source/installation/prep.rst b/doc/sphinx-guides/source/installation/prep.rst index 33b1aa3c7b5..abb4349d3ad 100644 --- a/doc/sphinx-guides/source/installation/prep.rst +++ b/doc/sphinx-guides/source/installation/prep.rst @@ -26,7 +26,7 @@ Advanced Installation There are some community-lead projects to use configuration management tools such as Ansible and Puppet to automate the installation and configuration of the Dataverse Software, but support for these solutions is limited to what the Dataverse Community can offer as described in each project's webpage: -- https://github.com/gdcc/dataverse-ansible +- https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible - https://gitlab.com/lip-computing/dataverse - https://github.com/IQSS/dataverse-puppet diff --git a/doc/sphinx-guides/source/qa/testing-approach.md b/doc/sphinx-guides/source/qa/testing-approach.md index 49b9075cf7f..817161d02a0 100644 --- a/doc/sphinx-guides/source/qa/testing-approach.md +++ b/doc/sphinx-guides/source/qa/testing-approach.md @@ -34,7 +34,7 @@ Think about risk. Is the feature or function part of a critical area such as per ## Smoke Test -1. Go to the homepage on (this server has production data). Scroll to the bottom to ensure the build number is the one you intend to test from Jenkins. +1. Go to the homepage on . Scroll to the bottom to ensure the build number is the one you intend to test from Jenkins. 1. Create a new user: It's fine to use a formulaic name with your initials and date and make the username and password the same, eg. kc080622. 1. Create a dataverse: You can use the same username. 1. Create a dataset: You can use the same username; fill in the required fields (do not use a template). diff --git a/makefile b/makefile index 1ffdb627275..315ff9c508c 100644 --- a/makefile +++ b/makefile @@ -1,5 +1,5 @@ -# We use "Sphinx=" to avoid packages like Sphinx-Substitution-Extensions -SPHINX_VERSION = $(shell grep "Sphinx=" ./doc/sphinx-guides/requirements.txt | awk -F'==' '{print $$2}') + +SPHINX_VERSION = $(shell grep "Sphinx" ./doc/sphinx-guides/requirements.txt | awk -F'==' '{print $$2}') docs-html: docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make clean && make html" @@ -11,4 +11,4 @@ docs-epub: docs-all: docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make clean && make html && make epub" - docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx-latexpdf:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make latexpdf LATEXMKOPTS=\"-interaction=nonstopmode\"; cd ../.. && ls -1 doc/sphinx-guides/build/latex/Dataverse.pdf" + docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx-latexpdf:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make latexpdf LATEXMKOPTS=\"-interaction=nonstopmode\"; cd ../.. && ls -1 doc/sphinx-guides/build/latex/Dataverse.pdf" \ No newline at end of file diff --git a/modules/container-base/README.md b/modules/container-base/README.md index 7a39457b766..f6854482073 100644 --- a/modules/container-base/README.md +++ b/modules/container-base/README.md @@ -25,7 +25,7 @@ provides in-depth information about content, building, tuning and so on for this **Where to get help and ask questions:** IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it. -You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community +You can join the Community Chat on Matrix at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community to ask for help and guidance. ## Supported Image Tags diff --git a/modules/container-configbaker/README.md b/modules/container-configbaker/README.md index 8b68e3db692..75862ee0809 100644 --- a/modules/container-configbaker/README.md +++ b/modules/container-configbaker/README.md @@ -19,7 +19,7 @@ provides information about this image. **Where to get help and ask questions:** IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it. -You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community +You can join the Community Chat on Matrix at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community to ask for help and guidance. ## Supported Image Tags diff --git a/modules/dataverse-spi/pom.xml b/modules/dataverse-spi/pom.xml index a603e274234..b00053fe5e0 100644 --- a/modules/dataverse-spi/pom.xml +++ b/modules/dataverse-spi/pom.xml @@ -13,7 +13,7 @@ io.gdcc dataverse-spi - 2.1.0${project.version.suffix} + 2.0.0${project.version.suffix} jar Dataverse SPI Plugin API @@ -64,13 +64,11 @@ - central - https://central.sonatype.com/repository/maven-snapshots/ + ossrh + https://s01.oss.sonatype.org/content/repositories/snapshots - ossrh - https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/ @@ -112,9 +110,7 @@ nexus-staging-maven-plugin true - ossrh - https://s01.oss.sonatype.org true diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java deleted file mode 100644 index 9478d39c4c2..00000000000 --- a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java +++ /dev/null @@ -1,61 +0,0 @@ -package io.gdcc.spi.export; - -/** - * - * @author landreev - * Provides an optional mechanism for defining various data retrieval options - * for the export subsystem in a way that should allow us adding support for - * more options going forward with minimal or no changes to the already - * implemented export plugins. - */ -public class ExportDataContext { - private boolean datasetMetadataOnly = false; - private boolean publicFilesOnly = false; - private Integer offset = null; - private Integer length = null; - - private ExportDataContext() { - - } - - public static ExportDataContext context() { - ExportDataContext context = new ExportDataContext(); - return context; - } - - public ExportDataContext withDatasetMetadataOnly() { - this.datasetMetadataOnly = true; - return this; - } - - public ExportDataContext withPublicFilesOnly() { - this.publicFilesOnly = true; - return this; - } - - public ExportDataContext withOffset(Integer offset) { - this.offset = offset; - return this; - } - - public ExportDataContext withLength(Integer length) { - this.length = length; - return this; - } - - public boolean isDatasetMetadataOnly() { - return datasetMetadataOnly; - } - - public boolean isPublicFilesOnly() { - return publicFilesOnly; - } - - public Integer getOffset() { - return offset; - } - - public Integer getLength() { - return length; - } -} diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java deleted file mode 100644 index 69f813f83ce..00000000000 --- a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java +++ /dev/null @@ -1,51 +0,0 @@ -package io.gdcc.spi.export; - -/** - * - * @author landreev - * Provides a mechanism for defining various data retrieval options for the - * export subsystem in a way that should allow us adding support for more - * options going forward with minimal or no changes to the existing code in - * export plugins. - */ -@Deprecated -public class ExportDataOption { - - public enum SupportedOptions { - DatasetMetadataOnly, - PublicFilesOnly; - } - - private SupportedOptions optionType; - - /*public static ExportDataOption addOption(String option) { - ExportDataOption ret = new ExportDataOption(); - - for (SupportedOptions supported : SupportedOptions.values()) { - if (supported.toString().equals(option)) { - ret.optionType = supported; - } - } - return ret; - }*/ - - public static ExportDataOption addDatasetMetadataOnly() { - ExportDataOption ret = new ExportDataOption(); - ret.optionType = SupportedOptions.DatasetMetadataOnly; - return ret; - } - - public static ExportDataOption addPublicFilesOnly() { - ExportDataOption ret = new ExportDataOption(); - ret.optionType = SupportedOptions.PublicFilesOnly; - return ret; - } - - public boolean isDatasetMetadataOnly() { - return SupportedOptions.DatasetMetadataOnly.equals(optionType); - } - - public boolean isPublicFilesOnly() { - return SupportedOptions.PublicFilesOnly.equals(optionType); - } -} diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java index 4197d978e79..d039ac39e8f 100644 --- a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java +++ b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java @@ -21,14 +21,8 @@ public interface ExportDataProvider { * OAI_ORE export are the only two that provide 'complete' * dataset-level metadata along with basic file metadata for each file * in the dataset. - * @param context - supplies optional parameters. Needs to support - * context.isDatasetMetadataOnly(). In a situation where we - * need to generate a format like DC that has no use for the - * file-level metadata, it makes sense to skip retrieving and - * formatting it, since there can be a very large number of - * files in a dataset. */ - JsonObject getDatasetJson(ExportDataContext... context); + JsonObject getDatasetJson(); /** * @@ -38,15 +32,14 @@ public interface ExportDataProvider { * @apiNote - THis, and the JSON format are the only two that provide complete * dataset-level metadata along with basic file metadata for each file * in the dataset. - * @param context - supplies optional parameters. */ - JsonObject getDatasetORE(ExportDataContext... context); + JsonObject getDatasetORE(); /** * Dataverse is capable of extracting DDI-centric metadata from tabular * datafiles. This detailed metadata, which is only available for successfully * "ingested" tabular files, is not included in the output of any other methods - * in this interface. + * in this interface. * * @return - a JSONArray with one entry per ingested tabular dataset file. * @apiNote - there is no JSON schema available for this output and the format @@ -54,26 +47,9 @@ public interface ExportDataProvider { * edu.harvard.iq.dataverse.export.DDIExporter and the @see * edu.harvard.iq.dataverse.util.json.JSONPrinter classes where this * output is used/generated (respectively). - * @param context - supplies optional parameters. */ - JsonArray getDatasetFileDetails(ExportDataContext... context); + JsonArray getDatasetFileDetails(); - /** - * Similar to the above, but - * a) retrieves the information for the ingested/tabular data files _only_ - * b) provides an option for retrieving this stuff in batches - * c) provides an option for skipping restricted/embargoed etc. files. - * Intended for datasets with massive numbers of tabular files and datavariables. - * @param context - supplies optional parameters. - * current (2.1.0) known use cases: - * context.isPublicFilesOnly(); - * context.getOffset(); - * context.getLength(); - * @return json array containing the datafile/filemetadata->datatable->datavariable metadata - * @throws ExportException - */ - JsonArray getTabularDataDetails(ExportDataContext ... context) throws ExportException; - /** * * @return - the subset of metadata conforming to the schema.org standard as @@ -82,9 +58,8 @@ public interface ExportDataProvider { * @apiNote - as this metadata export is not complete, it should only be used as * a starting point for an Exporter if it simplifies your exporter * relative to using the JSON or OAI_ORE exports. - * @param context - supplies optional parameters. */ - JsonObject getDatasetSchemaDotOrg(ExportDataContext... context); + JsonObject getDatasetSchemaDotOrg(); /** * @@ -93,9 +68,8 @@ public interface ExportDataProvider { * @apiNote - as this metadata export is not complete, it should only be used as * a starting point for an Exporter if it simplifies your exporter * relative to using the JSON or OAI_ORE exports. - * @param context - supplies optional parameters. */ - String getDataCiteXml(ExportDataContext... context); + String getDataCiteXml(); /** * If an Exporter has specified a prerequisite format name via the @@ -114,10 +88,9 @@ public interface ExportDataProvider { * malfunction, e.g. if you depend on format "ddi" and a third party * Exporter is configured to replace the internal ddi Exporter in * Dataverse. - * @param context - supplies optional parameters. */ - default Optional getPrerequisiteInputStream(ExportDataContext... context) { + default Optional getPrerequisiteInputStream() { return Optional.empty(); } - - } + +} diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java index 7132e74641b..1338a3c9734 100644 --- a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java +++ b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java @@ -85,6 +85,7 @@ default Optional getPrerequisiteFormatName() { return Optional.empty(); } + /** * Harvestable Exporters will be available as options in Dataverse's Harvesting mechanism. * @return true to make this exporter available as a harvesting option. diff --git a/src/main/docker/README.md b/src/main/docker/README.md index a32c91a810e..48416c196ca 100644 --- a/src/main/docker/README.md +++ b/src/main/docker/README.md @@ -24,7 +24,7 @@ for more details on tunable settings, locations, etc. **Where to get help and ask questions:** IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it. -You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community +You can join the Community Chat on Matrix at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community to ask for help and guidance. ## Supported Image Tags diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java index 28b061ffa2a..dec07a09643 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java @@ -9,7 +9,6 @@ import jakarta.persistence.Id; import jakarta.persistence.Index; import jakarta.persistence.JoinColumn; -import jakarta.persistence.NamedNativeQuery; import jakarta.persistence.NamedQueries; import jakarta.persistence.NamedQuery; import jakarta.persistence.OneToOne; @@ -36,18 +35,6 @@ @NamedQuery(name = "DatasetLinkingDataverse.findIdsByLinkingDataverseId", query = "SELECT o.dataset.id FROM DatasetLinkingDataverse AS o WHERE o.linkingDataverse.id = :linkingDataverseId") }) - - @NamedNativeQuery( - name = "DatasetLinkingDataverse.findByDatasetIdAndLinkingDataverseName", - query = """ - select o.linkingDataverse_id from DatasetLinkingDataverse as o - LEFT JOIN dataverse dv ON dv.id = o.linkingDataverse_id - WHERE o.dataset_id =? AND ((LOWER(dv.name) LIKE ? and ((SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?) - or (SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?))) - or (LOWER(dv.name) NOT LIKE ? and ((LOWER(dv.name) LIKE ?) - or (LOWER(dv.name) LIKE ?))))""" - ) - public class DatasetLinkingDataverse implements Serializable { private static final long serialVersionUID = 1L; @Id diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java index 6b0f8af6590..39c82bfa3f1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java @@ -5,7 +5,6 @@ */ package edu.harvard.iq.dataverse; -import jakarta.ejb.EJB; import java.util.ArrayList; import java.util.List; import java.util.logging.Logger; @@ -14,6 +13,7 @@ import jakarta.persistence.EntityManager; import jakarta.persistence.NoResultException; import jakarta.persistence.PersistenceContext; +import jakarta.persistence.Query; import jakarta.persistence.TypedQuery; /** @@ -28,9 +28,6 @@ public class DatasetLinkingServiceBean implements java.io.Serializable { @PersistenceContext(unitName = "VDCNet-ejbPU") private EntityManager em; - @EJB - DataverseServiceBean dataverseService; - public List findLinkedDatasets(Long dataverseId) { @@ -44,42 +41,13 @@ public List findLinkedDatasets(Long dataverseId) { } public List findLinkingDataverses(Long datasetId) { - return findLinkingDataverses(datasetId, ""); - } - - public List findLinkingDataverses(Long datasetId, String searchTerm) { List retList = new ArrayList<>(); - if (searchTerm == null || searchTerm.isEmpty()) { - TypedQuery typedQuery = em.createNamedQuery("DatasetLinkingDataverse.findByDatasetId", DatasetLinkingDataverse.class) - .setParameter("datasetId", datasetId); - for (DatasetLinkingDataverse datasetLinkingDataverse : typedQuery.getResultList()) { - retList.add(datasetLinkingDataverse.getLinkingDataverse()); - } - return retList; - - } else { - - String pattern = searchTerm.toLowerCase(); - - String pattern1 = pattern + "%"; - String pattern2 = "% " + pattern + "%"; - - // Adjust the queries for very short, 1 and 2-character patterns: - if (pattern.length() == 1) { - pattern1 = pattern; - pattern2 = pattern + " %"; - } - TypedQuery typedQuery - = em.createNamedQuery("DatasetLinkingDataverse.findByDatasetIdAndLinkingDataverseName", Long.class) - .setParameter(1, datasetId).setParameter(2, "%dataverse").setParameter(3, pattern1) - .setParameter(4, pattern2).setParameter(5, "%dataverse").setParameter(6, pattern1).setParameter(7, pattern2); - - for (Long id : typedQuery.getResultList()) { - retList.add(dataverseService.find(id)); - } - return retList; + TypedQuery typedQuery = em.createNamedQuery("DatasetLinkingDataverse.findByDatasetId", DatasetLinkingDataverse.class) + .setParameter("datasetId", datasetId); + for (DatasetLinkingDataverse datasetLinkingDataverse : typedQuery.getResultList()) { + retList.add(datasetLinkingDataverse.getLinkingDataverse()); } - + return retList; } public void save(DatasetLinkingDataverse datasetLinkingDataverse) { diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java index bf2326f0e06..3030922ea5e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java @@ -13,7 +13,6 @@ import jakarta.persistence.Id; import jakarta.persistence.Index; import jakarta.persistence.JoinColumn; -import jakarta.persistence.NamedNativeQuery; import jakarta.persistence.NamedQueries; import jakarta.persistence.NamedQuery; import jakarta.persistence.OneToOne; @@ -40,16 +39,6 @@ @NamedQuery(name = "DataverseLinkingDataverse.findIdsByLinkingDataverseId", query = "SELECT o.dataverse.id FROM DataverseLinkingDataverse AS o WHERE o.linkingDataverse.id = :linkingDataverseId") }) - @NamedNativeQuery( - name = "DataverseLinkingDataverse.findByDataverseIdAndLinkingDataverseName", - query = """ - select o.linkingDataverse_id from DataverseLinkingDataverse as o - LEFT JOIN dataverse dv ON dv.id = o.linkingDataverse_id - WHERE o.dataverse_id =? AND ((LOWER(dv.name) LIKE ? and ((SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?) - or (SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?))) - or (LOWER(dv.name) NOT LIKE ? and ((LOWER(dv.name) LIKE ?) - or (LOWER(dv.name) LIKE ?))))""" - ) public class DataverseLinkingDataverse implements Serializable { private static final long serialVersionUID = 1L; @Id diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java index 9f1bcde4c0e..834ff96e392 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java @@ -43,41 +43,12 @@ public List findLinkedDataverses(Long linkingDataverseId) { } public List findLinkingDataverses(Long dataverseId) { - - return findLinkingDataverses(dataverseId, ""); - } - - public List findLinkingDataverses(Long dataverseId, String searchTerm) { List retList = new ArrayList<>(); - if (searchTerm == null || searchTerm.isEmpty()) { - TypedQuery typedQuery = em.createNamedQuery("DataverseLinkingDataverse.findByDataverseId", DataverseLinkingDataverse.class) - .setParameter("dataverseId", dataverseId); - for (DataverseLinkingDataverse dataverseLinkingDataverse : typedQuery.getResultList()) { - retList.add(dataverseLinkingDataverse.getLinkingDataverse()); - } - - } else { - - String pattern = searchTerm.toLowerCase(); - - String pattern1 = pattern + "%"; - String pattern2 = "% " + pattern + "%"; - - // Adjust the queries for very short, 1 and 2-character patterns: - if (pattern.length() == 1) { - pattern1 = pattern; - pattern2 = pattern + " %"; - } - TypedQuery typedQuery - = em.createNamedQuery("DataverseLinkingDataverse.findByDataverseIdAndLinkingDataverseName", Long.class) - .setParameter(1, dataverseId).setParameter(2, "%dataverse").setParameter(3, pattern1) - .setParameter(4, pattern2).setParameter(5, "%dataverse").setParameter(6, pattern1).setParameter(7, pattern2); - - for (Long id : typedQuery.getResultList()) { - retList.add(dataverseService.find(id)); - } + TypedQuery typedQuery = em.createNamedQuery("DataverseLinkingDataverse.findByDataverseId", DataverseLinkingDataverse.class) + .setParameter("dataverseId", dataverseId); + for (DataverseLinkingDataverse dataverseLinkingDataverse : typedQuery.getResultList()) { + retList.add(dataverseLinkingDataverse.getLinkingDataverse()); } - return retList; } diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java index 57bc0bc5450..c14711060af 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java @@ -505,28 +505,17 @@ public List filterByAliasQuery(String filterQuery) { return ret; } - public List filterDataversesForLinking(String query, DataverseRequest req, DvObject dvo) { + public List filterDataversesForLinking(String query, DataverseRequest req, Dataset dataset) { List dataverseList = new ArrayList<>(); List results = filterDataversesByNamePattern(query); - - if (results == null || results.isEmpty()) { - return null; - } - - Dataset linkedDataset = null; - Dataverse linkedDataverse = null; - List alreadyLinkeddv_ids; - - if ((dvo instanceof Dataset)) { - linkedDataset = (Dataset) dvo; - alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + linkedDataset.getId()).getResultList(); - } else { - linkedDataverse = (Dataverse) dvo; - alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM dataverselinkingdataverse WHERE dataverse_id = " + linkedDataverse.getId()).getResultList(); + + if (results == null || results.size() == 0) { + return null; } + List alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + dataset.getId()).getResultList(); List remove = new ArrayList<>(); if (alreadyLinkeddv_ids != null && !alreadyLinkeddv_ids.isEmpty()) { @@ -534,15 +523,10 @@ public List filterDataversesForLinking(String query, DataverseRequest remove.add(removeIt); }); } - - if (dvo instanceof Dataverse dataverse) { - remove.add(dataverse); - } - + for (Dataverse res : results) { if (!remove.contains(res)) { - if ((linkedDataset != null && this.permissionService.requestOn(req, res).has(Permission.LinkDataset)) - || (linkedDataverse != null && this.permissionService.requestOn(req, res).has(Permission.LinkDataverse))) { + if (this.permissionService.requestOn(req, res).has(Permission.LinkDataset)) { dataverseList.add(res); } } @@ -550,54 +534,6 @@ public List filterDataversesForLinking(String query, DataverseRequest return dataverseList; } - - public List removeUnlinkableDataverses(List allWithPerms, DvObject dvo) { - List dataverseList = new ArrayList<>(); - Dataset linkedDataset = null; - Dataverse linkedDataverse = null; - List alreadyLinkeddv_ids; - - if ((dvo instanceof Dataset)) { - linkedDataset = (Dataset) dvo; - alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + linkedDataset.getId()).getResultList(); - } else { - linkedDataverse = (Dataverse) dvo; - alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM dataverselinkingdataverse WHERE dataverse_id = " + linkedDataverse.getId()).getResultList(); - } - - List remove = new ArrayList<>(); - - if (alreadyLinkeddv_ids != null && !alreadyLinkeddv_ids.isEmpty()) { - alreadyLinkeddv_ids.stream().map((testDVId) -> this.find(testDVId)).forEachOrdered((removeIt) -> { - remove.add(removeIt); - }); - } - - - if (dvo instanceof Dataverse dataverse) { - remove.add(dataverse); - } - - DvObject testDVO = dvo; - //Remove DVO's parent up to Root - while (testDVO != null) { - if (testDVO.getOwner() == null) { - break; // we are at the root; which by definition is metadata block root, regardless of the value - } - remove.add((Dataverse) testDVO.getOwner()); - testDVO = testDVO.getOwner(); - } - - for (Dataverse res : allWithPerms) { - if (!remove.contains(res)) { - dataverseList.add(res); - } - } - - return dataverseList; - } - - public List filterDataversesForUnLinking(String query, DataverseRequest req, Dataset dataset) { List alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + dataset.getId()).getResultList(); List dataverseList = new ArrayList<>(); diff --git a/src/main/java/edu/harvard/iq/dataverse/FilePage.java b/src/main/java/edu/harvard/iq/dataverse/FilePage.java index dd1bd56d5bd..50ebbc33634 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FilePage.java +++ b/src/main/java/edu/harvard/iq/dataverse/FilePage.java @@ -420,11 +420,6 @@ public boolean isGuestbookPopupRequiredAtDownload(){ DatasetVersion workingVersion = fileMetadata.getDatasetVersion(); return FileUtil.isGuestbookPopupRequired(workingVersion) && !workingVersion.getDataset().getEffectiveGuestbookEntryAtRequest(); } - - public boolean isGuestbookPopupRequired(){ - DatasetVersion workingVersion = fileMetadata.getDatasetVersion(); - return FileUtil.isGuestbookPopupRequired(workingVersion); - } public void setFileMetadata(FileMetadata fileMetadata) { this.fileMetadata = fileMetadata; diff --git a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java index d492991bb62..e00e303356e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java @@ -96,7 +96,7 @@ public class PermissionServiceBean { DatasetVersionFilesServiceBean datasetVersionFilesServiceBean; private static final String LIST_ALL_DATAVERSES_SUPERUSER_HAS_PERMISSION = """ - SELECT id, name, alias FROM DATAVERSE dv + SELECT id, name, alias FROM DATAVERSE """; private static final String LIST_ALL_DATAVERSES_USER_HAS_PERMISSION = """ @@ -105,7 +105,7 @@ WITH grouplist AS ( WHERE explicitgroup_authenticateduser.containedauthenticatedusers_id = @USERID ) - SELECT * FROM DATAVERSE dv WHERE id IN ( + SELECT * FROM DATAVERSE WHERE id IN ( SELECT definitionpoint_id FROM roleassignment WHERE roleassignment.assigneeidentifier IN ( @@ -161,24 +161,8 @@ AND EXISTS (SELECT id FROM dataverserole WHERE dataverserole.id = roleassignment AND @IPRANGESQL ) ) - ) + ) """; - - private static final String AND = """ - and - """; - - private static final String WHERE = """ - where - """; - - private static final String SEARCH_PARAMS = """ - ((LOWER(dv.name) LIKE ? and ((SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?) - or (SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?))) - or (LOWER(dv.name) NOT LIKE ? and ((LOWER(dv.name) LIKE ?) - or (LOWER(dv.name) LIKE ?)))) - """; - /** * A request-level permission query (e.g includes IP ras). */ @@ -937,21 +921,12 @@ private boolean hasUnrestrictedReleasedFiles(DatasetVersion targetDatasetVersion Long result = em.createQuery(criteriaQuery).getSingleResult(); return result > 0; } - + public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, Permission permission) { - return findPermittedCollections(request, user, 1 << permission.ordinal(), ""); + return findPermittedCollections(request, user, 1 << permission.ordinal()); } - public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, Permission permission, String searchTerm) { - return findPermittedCollections(request, user, 1 << permission.ordinal(), searchTerm); - } - public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, int permissionBit) { - return findPermittedCollections(request, user, permissionBit, ""); - } - - - public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, int permissionBit, String searchTerm) { if (user != null) { // IP Group - Only check IP if a User is calling for themself String ipRangeSQL = "FALSE"; @@ -983,64 +958,13 @@ public List findPermittedCollections(DataverseRequest request, Authen String sqlCode; if (user.isSuperuser()) { sqlCode = LIST_ALL_DATAVERSES_SUPERUSER_HAS_PERMISSION; - - if (searchTerm == null || searchTerm.isEmpty()) { - return em.createNativeQuery(sqlCode, Dataverse.class).getResultList(); - } else { - sqlCode = LIST_ALL_DATAVERSES_SUPERUSER_HAS_PERMISSION.concat(WHERE).concat(SEARCH_PARAMS); - - String pattern = searchTerm.toLowerCase(); - String pattern1 = pattern + "%"; - String pattern2 = "% " + pattern + "%"; - - // Adjust the queries for very short, 1 - if (pattern.length() == 1) { - pattern1 = pattern; - pattern2 = pattern + " %"; - } - Query query = em.createNativeQuery(sqlCode, Dataverse.class); - query.setParameter(1, "%dataverse"); - query.setParameter(2, pattern1); - query.setParameter(3, pattern2); - query.setParameter(4, "%dataverse"); - query.setParameter(5, pattern1); - query.setParameter(6, pattern2); - return query.getResultList(); - - } } else { - if (searchTerm == null || searchTerm.isEmpty()) { - sqlCode = LIST_ALL_DATAVERSES_USER_HAS_PERMISSION - .replace("@USERID", String.valueOf(user.getId())) - .replace("@PERMISSIONBIT", String.valueOf(permissionBit)) - .replace("@IPRANGESQL", ipRangeSQL); - return em.createNativeQuery(sqlCode, Dataverse.class).getResultList(); - } else { - String pattern = searchTerm.toLowerCase(); - String pattern1 = pattern + "%"; - String pattern2 = "% " + pattern + "%"; - - // Adjust the queries for very short, 1 - if (pattern.length() == 1) { - pattern1 = pattern; - pattern2 = pattern + " %"; - } - - sqlCode = LIST_ALL_DATAVERSES_USER_HAS_PERMISSION.concat(AND).concat(SEARCH_PARAMS) - .replace("@USERID", String.valueOf(user.getId())) - .replace("@PERMISSIONBIT", String.valueOf(permissionBit)) - .replace("@IPRANGESQL", ipRangeSQL); - - Query query = em.createNativeQuery(sqlCode, Dataverse.class); - query.setParameter(1, "%dataverse"); - query.setParameter(2, pattern1); - query.setParameter(3, pattern2); - query.setParameter(4, "%dataverse"); - query.setParameter(5, pattern1); - query.setParameter(6, pattern2); - return query.getResultList(); - } + sqlCode = LIST_ALL_DATAVERSES_USER_HAS_PERMISSION + .replace("@USERID", String.valueOf(user.getId())) + .replace("@PERMISSIONBIT", String.valueOf(permissionBit)) + .replace("@IPRANGESQL", ipRangeSQL); } + return em.createNativeQuery(sqlCode, Dataverse.class).getResultList(); } return null; } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java b/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java index 0ce84844289..46e8263da15 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java @@ -615,12 +615,11 @@ protected DvObject findDvo(@NotNull final String id) throws WrappedResponse { * * @param dvIdtf * @param type - * @param testForReleased * @return DvObject if type matches or throw exception * @throws WrappedResponse */ @NotNull - protected DvObject findDvoByIdAndTypeOrDie(@NotNull final String dvIdtf, String type, boolean testForReleased) throws WrappedResponse { + protected DvObject findDvoByIdAndFeaturedItemTypeOrDie(@NotNull final String dvIdtf, String type) throws WrappedResponse { try { DataverseFeaturedItem.TYPES dvType = DataverseFeaturedItem.getDvType(type); DvObject dvObject = null; @@ -653,12 +652,7 @@ protected DvObject findDvoByIdAndTypeOrDie(@NotNull final String dvIdtf, String } } } - if (testForReleased){ - DataverseFeaturedItem.validateTypeAndDvObject(dvIdtf, dvObject, dvType); - } - if (dvObject == null) { - throw new WrappedResponse(notFound(BundleUtil.getStringFromBundle("find.dvo.error.dvObjectNotFound", Collections.singletonList(dvIdtf)))); - } + DataverseFeaturedItem.validateTypeAndDvObject(dvIdtf, dvObject, dvType); return dvObject; } catch (IllegalArgumentException e) { throw new WrappedResponse(error(Response.Status.BAD_REQUEST, e.getMessage())); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java index 12989a1b70c..d55f582ecae 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java @@ -20,7 +20,6 @@ import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.util.StringUtil; import edu.harvard.iq.dataverse.util.cache.CacheFactoryBean; -import edu.harvard.iq.dataverse.util.json.JsonPrinter; import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; import edu.harvard.iq.dataverse.validation.EMailValidator; import edu.harvard.iq.dataverse.EjbDataverseEngine; @@ -2181,8 +2180,7 @@ public Response addRoleAssignementsToChildren(@Context ContainerRequestContext c @GET @AuthRequired @Path("/dataverse/{alias}/storageDriver") - public Response getStorageDriver(@Context ContainerRequestContext crc, @PathParam("alias") String alias, - @QueryParam("getEffective") Boolean getEffective) throws WrappedResponse { + public Response getStorageDriver(@Context ContainerRequestContext crc, @PathParam("alias") String alias) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); @@ -2195,12 +2193,8 @@ public Response getStorageDriver(@Context ContainerRequestContext crc, @PathPara } catch (WrappedResponse wr) { return wr.getResponse(); } - - if (getEffective != null && getEffective) { - return ok(JsonPrinter.jsonStorageDriver(dataverse.getEffectiveStorageDriverId(), null)); - } else { - return ok(JsonPrinter.jsonStorageDriver(dataverse.getStorageDriverId(), null)); - } + //Note that this returns what's set directly on this dataverse. If null/DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER, the user would have to recurse the chain of parents to find the effective storageDriver + return ok(dataverse.getStorageDriverId()); } @PUT diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index aa39d1b6c94..10abdb63d82 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -2608,7 +2608,7 @@ public Response getCurationStatus(@Context ContainerRequestContext crc, canSeeStatus = permissionSvc.requestOn(createDataverseRequest(user), ds).has(Permission.PublishDataset); } - if (canSeeStatus) { + if (dsv.isDraft() && (canSeeStatus)) { List statuses = includeHistory ? dsv.getCurationStatuses() : Collections.singletonList(dsv.getCurrentCurationStatus()); if (includeHistory) { JsonArrayBuilder arrayBuilder = Json.createArrayBuilder(); @@ -3689,7 +3689,7 @@ public Response getFileStore(@Context ContainerRequestContext crc, @PathParam("i return error(Response.Status.NOT_FOUND, "No such dataset"); } - return ok(JsonPrinter.jsonStorageDriver(dataset.getEffectiveStorageDriverId(), dataset)); + return response(req -> ok(dataset.getEffectiveStorageDriverId()), getRequestUser(crc)); } @PUT diff --git a/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java b/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java index 7fbdd79e3c3..30c3146fbfb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java @@ -63,7 +63,7 @@ public Response updateFeaturedItem(@Context ContainerRequestContext crc, if (dataverseFeaturedItem == null) { throw new WrappedResponse(error(Response.Status.NOT_FOUND, MessageFormat.format(BundleUtil.getStringFromBundle("dataverseFeaturedItems.errors.notFound"), id))); } - DvObject dvObject = (dvObjectIdtf != null) ? findDvoByIdAndTypeOrDie(dvObjectIdtf, type, true) : null; + DvObject dvObject = (dvObjectIdtf != null) ? findDvoByIdAndFeaturedItemTypeOrDie(dvObjectIdtf, type) : null; UpdatedDataverseFeaturedItemDTO updatedDataverseFeaturedItemDTO = UpdatedDataverseFeaturedItemDTO.fromFormData(content, displayOrder, keepFile, imageFileInputStream, contentDispositionHeader, type, dvObject); return ok(json(execCommand(new UpdateDataverseFeaturedItemCommand(createDataverseRequest(getRequestUser(crc)), dataverseFeaturedItem, updatedDataverseFeaturedItemDTO)))); } catch (WrappedResponse e) { diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java index 7322a8c9341..ae82ff46522 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java @@ -15,7 +15,6 @@ import edu.harvard.iq.dataverse.authorization.groups.impl.explicit.ExplicitGroup; import edu.harvard.iq.dataverse.authorization.groups.impl.explicit.ExplicitGroupProvider; import edu.harvard.iq.dataverse.authorization.groups.impl.explicit.ExplicitGroupServiceBean; -import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.dataset.DatasetType; @@ -120,7 +119,7 @@ public class Dataverses extends AbstractApiBean { @EJB DataverseFeaturedItemServiceBean dataverseFeaturedItemServiceBean; - + @POST @AuthRequired public Response addRoot(@Context ContainerRequestContext crc, String body) { @@ -715,7 +714,7 @@ public Response getDataverse(@Context ContainerRequestContext crc, @PathParam("i return response(req -> { Dataverse dataverse = execCommand(new GetDataverseCommand(req, findDataverseOrDie(idtf))); boolean hideEmail = settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false); - return ok(json(dataverse, hideEmail, returnOwners, false, returnChildCount ? dataverseService.getChildCount(dataverse) : null)); + return ok(json(dataverse, hideEmail, returnOwners, returnChildCount ? dataverseService.getChildCount(dataverse) : null)); }, getRequestUser(crc)); } @@ -1772,36 +1771,6 @@ public Response linkDataverse(@Context ContainerRequestContext crc, @PathParam(" return ex.getResponse(); } } - - @GET - @AuthRequired - @Produces(MediaType.APPLICATION_JSON) - @Path("{identifier}/{type}/linkingDataverses") - public Response getLinkingDataverseList(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @QueryParam("searchTerm") String searchTerm, @QueryParam("alreadyLinking") boolean alreadyLinking, @PathParam("type") String type) { - - try { - - DvObject dvObject = findDvoByIdAndTypeOrDie(dvIdtf, type, false); - List dataversesForLinking = execCommand(new GetLinkingDataverseListCommand( - createDataverseRequest(getRequestUser(crc)), - dvObject, - searchTerm, - alreadyLinking - )); - - JsonArrayBuilder dvBuilder = Json.createArrayBuilder(); - if (dataversesForLinking != null && !dataversesForLinking.isEmpty()) { - for (Dataverse dv : dataversesForLinking) { - dvBuilder.add(json(dv, true)); - } - } - return ok(dvBuilder); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } - } - - @GET @AuthRequired @@ -1842,7 +1811,7 @@ public Response createFeaturedItem(@Context ContainerRequestContext crc, try { dataverse = findDataverseOrDie(dvIdtf); if (dvObjectIdtf != null) { - dvObject = findDvoByIdAndTypeOrDie(dvObjectIdtf, type, true); + dvObject = findDvoByIdAndFeaturedItemTypeOrDie(dvObjectIdtf, type); } } catch (WrappedResponse wr) { return wr.getResponse(); @@ -1930,7 +1899,7 @@ public Response updateFeaturedItems( // ignore dvObject if the id is missing or an empty string DvObject dvObject = dvObjectIdtf.get(i) != null && !dvObjectIdtf.get(i).isEmpty() - ? findDvoByIdAndTypeOrDie(dvObjectIdtf.get(i), types.get(i), true) : null; + ? findDvoByIdAndFeaturedItemTypeOrDie(dvObjectIdtf.get(i), types.get(i)) : null; if (ids.get(i) == 0) { newItems.add(NewDataverseFeaturedItemDTO.fromFormData( contents.get(i), displayOrders.get(i), fileInputStream, contentDisposition, types.get(i), dvObject)); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java b/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java index ca4f55da822..562fd7fcb81 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java @@ -25,15 +25,9 @@ import java.net.URL; import java.util.Iterator; import java.util.List; -import java.util.concurrent.Future; -import java.util.concurrent.RejectedExecutionException; -import java.util.concurrent.atomic.AtomicLong; import java.util.logging.Level; import java.util.logging.Logger; - -import jakarta.annotation.Resource; import jakarta.ejb.EJB; -import jakarta.enterprise.concurrent.ManagedExecutorService; import jakarta.json.Json; import jakarta.json.JsonArray; import jakarta.json.JsonArrayBuilder; @@ -68,13 +62,6 @@ public class MakeDataCountApi extends AbstractApiBean { @EJB SystemConfig systemConfig; - // Inject the managed executor service provided by the container - @Resource(name = "concurrent/CitationUpdateExecutor") - private ManagedExecutorService executorService; - - // Track the last execution time to implement rate limiting during Citation updates - private static final AtomicLong lastExecutionTime = new AtomicLong(0); - /** * TODO: For each dataset, send the following: * @@ -154,109 +141,34 @@ public Response addUsageMetricsFromSushiReportAll(@QueryParam("reportOnDisk") St @POST @Path("{id}/updateCitationsForDataset") - public Response updateCitationsForDataset(@PathParam("id") String id) { + public Response updateCitationsForDataset(@PathParam("id") String id) throws IOException { try { - // First validate that the dataset exists and has a valid DOI - final Dataset dataset = findDatasetOrDie(id); - final GlobalId pid = dataset.getGlobalId(); - final PidProvider pidProvider = PidUtil.getPidProvider(pid.getProviderId()); - + Dataset dataset = findDatasetOrDie(id); + GlobalId pid = dataset.getGlobalId(); + PidProvider pidProvider = PidUtil.getPidProvider(pid.getProviderId()); // Only supported for DOIs and for DataCite DOI providers - if (!DataCiteDOIProvider.TYPE.equals(pidProvider.getProviderType())) { + if(!DataCiteDOIProvider.TYPE.equals(pidProvider.getProviderType())) { return error(Status.BAD_REQUEST, "Only DataCite DOI providers are supported"); } + String persistentId = pid.toString(); - // Submit the task to the managed executor service - Future future; + // DataCite wants "doi=", not "doi:". + String authorityPlusIdentifier = persistentId.replaceFirst("doi:", ""); + // Request max page size and then loop to handle multiple pages + URL url = null; try { - future = executorService.submit(() -> { - try { - // Apply rate limiting if enabled - applyRateLimit(); - - // Process the citation update - boolean success = processCitationUpdate(dataset, pid, pidProvider); - - // Update the last execution time after processing - lastExecutionTime.set(System.currentTimeMillis()); - - if (success) { - logger.fine("Successfully processed citation update for dataset " + id); - } else { - logger.warning("Failed to process citation update for dataset " + id); - } - } catch (Exception e) { - logger.log(Level.SEVERE, "Error processing citation update for dataset " + id, e); - } - }); - - JsonObjectBuilder output = Json.createObjectBuilder(); - output.add("status", "queued"); - output.add("message", "Citation update for dataset " + id + " has been queued for processing"); - return ok(output); - } catch (RejectedExecutionException ree) { - logger.warning("Citation update for dataset " + id + " was rejected: Queue is full"); - return error(Status.SERVICE_UNAVAILABLE, - "Citation update service is currently at capacity. Please try again later."); + url = new URI(JvmSettings.DATACITE_REST_API_URL.lookup(pidProvider.getId()) + + "/events?doi=" + + authorityPlusIdentifier + + "&source=crossref&page[size]=1000&page[cursor]=1").toURL(); + } catch (URISyntaxException e) { + //Nominally this means a config error/ bad DATACITE_REST_API_URL for this provider + logger.warning("Unable to create URL for " + persistentId + ", pidProvider " + pidProvider.getId()); + return error(Status.INTERNAL_SERVER_ERROR, "Unable to create DataCite URL to retrieve citations."); } - } catch (WrappedResponse wr) { - return wr.getResponse(); - } - } - - /** - * Apply rate limiting by waiting if necessary - */ - private void applyRateLimit() { - // Check if rate limiting is enabled - long minDelay = JvmSettings.API_MDC_UPDATE_MIN_DELAY_MS.lookupOptional(Long.class).orElse(0l); - if(minDelay ==0) { - return; - } - // Calculate how long to wait - long lastExecution = lastExecutionTime.get(); - long currentTime = System.currentTimeMillis(); - long elapsedTime = currentTime - lastExecution; - - // If not enough time has passed since the last execution, wait - if (lastExecution > 0 && elapsedTime < minDelay) { - long waitTime = minDelay - elapsedTime; - logger.fine("Rate limiting: waiting " + waitTime + " ms before processing next citation update"); - try { - Thread.sleep(waitTime); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - logger.warning("Rate limiting sleep interrupted: " + e.getMessage()); - } - } - } - - /** - * Process the citation update for a dataset - * This method contains the logic that was previously in updateCitationsForDataset - * @return true if processing was successful, false otherwise - */ - private boolean processCitationUpdate(Dataset dataset, GlobalId pid, PidProvider pidProvider) { - String persistentId = pid.asRawIdentifier(); - - // Request max page size and then loop to handle multiple pages - URL url = null; - try { - url = new URI(JvmSettings.DATACITE_REST_API_URL.lookup(pidProvider.getId()) + - "/events?doi=" + - persistentId + - "&source=crossref&page[size]=1000&page[cursor]=1").toURL(); - } catch (URISyntaxException | MalformedURLException e) { - //Nominally this means a config error/ bad DATACITE_REST_API_URL for this provider - logger.warning("Unable to create URL for " + persistentId + ", pidProvider " + pidProvider.getId()); - return false; - } - - logger.fine("Retrieving Citations from " + url.toString()); - boolean nextPage = true; - JsonArrayBuilder dataBuilder = Json.createArrayBuilder(); - - try { + logger.fine("Retrieving Citations from " + url.toString()); + boolean nextPage = true; + JsonArrayBuilder dataBuilder = Json.createArrayBuilder(); do { HttpURLConnection connection = (HttpURLConnection) url.openConnection(); connection.setRequestMethod("GET"); @@ -264,56 +176,34 @@ private boolean processCitationUpdate(Dataset dataset, GlobalId pid, PidProvider if (status != 200) { logger.warning("Failed to get citations from " + url.toString()); connection.disconnect(); - return false; + return error(Status.fromStatusCode(status), "Failed to get citations from " + url.toString()); } - JsonObject report; try (InputStream inStream = connection.getInputStream()) { report = JsonUtil.getJsonObject(inStream); } finally { connection.disconnect(); } - JsonObject links = report.getJsonObject("links"); JsonArray data = report.getJsonArray("data"); Iterator iter = data.iterator(); while (iter.hasNext()) { - JsonValue citationValue = iter.next(); - JsonObject citation = (JsonObject) citationValue; - - // Filter out relations we don't use (e.g. hasPart) to lower memory req. with many files - if (citation.containsKey("attributes")) { - JsonObject attributes = citation.getJsonObject("attributes"); - if (attributes.containsKey("relation-type-id")) { - String relationshipType = attributes.getString("relation-type-id"); - - // Only add citations with relationship types we care about - if (DatasetExternalCitationsServiceBean.inboundRelationships.contains(relationshipType) || - DatasetExternalCitationsServiceBean.outboundRelationships.contains(relationshipType)) { - dataBuilder.add(citationValue); - } - } - } + dataBuilder.add(iter.next()); } - if (links.containsKey("next")) { try { url = new URI(links.getString("next")).toURL(); - applyRateLimit(); } catch (URISyntaxException e) { logger.warning("Unable to create URL from DataCite response: " + links.getString("next")); - return false; + return error(Status.INTERNAL_SERVER_ERROR, "Unable to retrieve all results from DataCite"); } } else { nextPage = false; } - logger.fine("body of citation response: " + report.toString()); } while (nextPage == true); - JsonArray allData = dataBuilder.build(); List datasetExternalCitations = datasetExternalCitationsService.parseCitations(allData); - /* * ToDo: If this is the only source of citations, we should remove all the existing ones for the dataset and repopulate them. * As is, this call doesn't remove old citations if there are now none (legacy issue if we decide to stop counting certain types of citation @@ -326,16 +216,14 @@ private boolean processCitationUpdate(Dataset dataset, GlobalId pid, PidProvider datasetExternalCitationsService.save(dm); } } - - logger.fine("Citation update completed for dataset " + dataset.getId() + - " with " + datasetExternalCitations.size() + " citations"); - return true; - } catch (IOException e) { - logger.log(Level.WARNING, "Error processing citation update for dataset " + dataset.getId(), e); - return false; + + JsonObjectBuilder output = Json.createObjectBuilder(); + output.add("citationCount", datasetExternalCitations.size()); + return ok(output); + } catch (WrappedResponse wr) { + return wr.getResponse(); } } - @GET @Path("{yearMonth}/processingState") public Response getProcessingState(@PathParam("yearMonth") String yearMonth) { diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java index 0a0d154a64e..22a3d825cfb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java @@ -52,7 +52,7 @@ public class ShibServiceBean { private static final String INCOMMON_MDQ_API_BASE = "https://mdq.incommon.org"; private static final String INCOMMON_MDQ_API_ENTITIES_URL = INCOMMON_MDQ_API_BASE + "/entities/"; private static final String INCOMMON_WAYFINDER_URL = "https://wayfinder.incommon.org"; - private static final String INCOMMON_ORGANIZATION_ATTRIBUTE = "OrganizationDisplayName"; + /** * "Production" means "don't mess with the HTTP request". */ @@ -233,7 +233,7 @@ public String getAffiliationViaMDQ(String shibIdp) { if (event == XMLStreamConstants.START_ELEMENT) { String currentElement = xmlr.getLocalName(); - if (INCOMMON_ORGANIZATION_ATTRIBUTE.equals(currentElement)) { + if ("".equals(currentElement)) { int eventType = xmlr.next(); if (eventType == XMLStreamConstants.CHARACTERS) { String affiliation = xmlr.getText(); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java index e4fd5373c7d..8227572da3b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java @@ -105,14 +105,10 @@ private void processInputLevels(CommandContext ctxt) { ctxt.fieldTypeInputLevels().deleteDataverseFieldTypeInputLevelFor(dataverse); } else { dataverse.addInputLevelsMetadataBlocksIfNotPresent(inputLevels); - //if levels not empty either create or update (handled by save - update when id not null create if null) + ctxt.fieldTypeInputLevels().deleteDataverseFieldTypeInputLevelFor(dataverse); inputLevels.forEach(inputLevel -> { - DataverseFieldTypeInputLevel ftil = ctxt.fieldTypeInputLevels().findByDataverseIdDatasetFieldTypeId(dataverse.getId(), inputLevel.getDatasetFieldType().getId()); - if(ftil != null){ - inputLevel.setId(ftil.getId()); - } inputLevel.setDataverse(dataverse); - ctxt.fieldTypeInputLevels().save(inputLevel); + ctxt.fieldTypeInputLevels().create(inputLevel); }); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java index 9ebdd28e009..3629432b7e4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java @@ -15,7 +15,6 @@ import edu.harvard.iq.dataverse.DatasetField; import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.TermsOfUseAndAccess; -import edu.harvard.iq.dataverse.CurationStatus; import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.RoleAssignment; @@ -28,8 +27,6 @@ import java.util.logging.Level; import java.util.logging.Logger; -import org.apache.commons.lang3.StringUtils; - /** * * @author qqmyers @@ -76,7 +73,7 @@ public Dataset execute(CommandContext ctxt) throws CommandException { newTerms.setDatasetVersion(updateVersion); updateVersion.setTermsOfUseAndAccess(newTerms); - //Version Note + //Creation Note updateVersion.setVersionNote(newVersion.getVersionNote()); // Clear unnecessary terms relationships .... @@ -99,24 +96,6 @@ public Dataset execute(CommandContext ctxt) throws CommandException { updateVersion.getWorkflowComments().addAll(newComments); } - // Transfer curation status entries from draft to published version - List draftCurationStatuses = newVersion.getCurationStatuses(); - if (draftCurationStatuses != null && !draftCurationStatuses.isEmpty()) { - for (CurationStatus cs : draftCurationStatuses) { - // Update the dataset version reference - //This call sets the version in the curationstatus object as well - updateVersion.addCurationStatus(cs); - } - // Clear the list from the draft version - newVersion.getCurationStatuses().clear(); - } - - // Add a new empty curation status to clear the status in the published version (as done in the FinalizeDatasetPublicationCommand) - CurationStatus status = updateVersion.getCurrentCurationStatus(); - if (status != null && StringUtils.isNotBlank(status.getLabel())) { - updateVersion.addCurationStatus(new CurationStatus(null, updateVersion, getRequest().getAuthenticatedUser())); - } - // we have to merge to update the database but not flush because // we don't want to create two draft versions! Dataset tempDataset = getDataset(); @@ -222,7 +201,7 @@ public Dataset execute(CommandContext ctxt) throws CommandException { // This can be corrected by running the update PID API later, but who will look in the log? // With the change to not use the DeleteDatasetVersionCommand above and other // fixes, this error may now cleanly restore the initial state - // with the draft and last published versions unchanged, but this has not yet been tested. + // with the draft and last published versions unchanged, but this has not yet bee tested. // (Alternately this could move to onSuccess if we intend it to stay non-fatal.) logger.log(Level.WARNING, "Curate Published DatasetVersion: exception while updating PID metadata:{0}", ex.getMessage()); } diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java deleted file mode 100644 index 466871b88e4..00000000000 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java +++ /dev/null @@ -1,113 +0,0 @@ - -package edu.harvard.iq.dataverse.engine.command.impl; - -import edu.harvard.iq.dataverse.Dataset; -import edu.harvard.iq.dataverse.Dataverse; -import edu.harvard.iq.dataverse.DvObject; -import edu.harvard.iq.dataverse.authorization.Permission; -import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; -import edu.harvard.iq.dataverse.authorization.users.User; -import edu.harvard.iq.dataverse.engine.command.AbstractCommand; -import edu.harvard.iq.dataverse.engine.command.CommandContext; -import edu.harvard.iq.dataverse.engine.command.DataverseRequest; -import edu.harvard.iq.dataverse.engine.command.RequiredPermissions; -import edu.harvard.iq.dataverse.engine.command.exception.CommandException; -import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; -import edu.harvard.iq.dataverse.util.BundleUtil; -import java.util.ArrayList; -import java.util.EnumSet; -import java.util.List; - -/** - * - * @author stephenkraffmiller - */ -@RequiredPermissions({}) -public class GetLinkingDataverseListCommand extends AbstractCommand> { - - private final String searchTerm; - private final DvObject dvObject; - private final boolean alreadyLinked; - - public GetLinkingDataverseListCommand(DataverseRequest aRequest, DvObject dvObject, String searchTerm) { - super(aRequest, dvObject); - this.searchTerm = searchTerm; - this.dvObject = dvObject; - this.alreadyLinked = false; - } - - public GetLinkingDataverseListCommand(DataverseRequest aRequest, DvObject dvObject, boolean alreadyLinked) { - super(aRequest, dvObject); - this.searchTerm = ""; - this.dvObject = dvObject; - this.alreadyLinked = alreadyLinked; - } - - public GetLinkingDataverseListCommand(DataverseRequest aRequest, DvObject dvObject, String searchTerm, boolean alreadyLinked) { - super(aRequest, dvObject); - this.searchTerm = searchTerm; - this.dvObject = dvObject; - this.alreadyLinked = alreadyLinked; - } - - - - @Override - public List execute(CommandContext ctxt) throws CommandException { - - User requestUser = (User) getRequest().getUser(); - AuthenticatedUser authUser; - if (!requestUser.isAuthenticated()) { - throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataverse.link.user"), this); - } else { - authUser = (AuthenticatedUser) requestUser; - } - List dataversesForLinking; - String searchParam; - if (searchTerm != null) { - searchParam = searchTerm; - } else { - searchParam = ""; - } - - //Find Permitted Collections now takes a Search Term to filter down collections the user may link - Permission permToCheck; - if (dvObject instanceof Dataset) { - permToCheck = Permission.LinkDataset; - } else { - permToCheck = Permission.LinkDataverse; - } - //depending on the already linked boolean the command will return a list of Dataverses available for linking - // or a list of dataverses to which the object has already been linked - for the unlink function - if (!alreadyLinked) { - dataversesForLinking = ctxt.permissions().findPermittedCollections(getRequest(), authUser, permToCheck, searchParam); - //Don't bother with checking for already linked if there are none to be tested. - if (dataversesForLinking == null || dataversesForLinking.isEmpty()) { - return dataversesForLinking; - } - return ctxt.dataverses().removeUnlinkableDataverses(dataversesForLinking, dvObject); - } else { - List dataversesAlreadyLinked; - List dataversesAlreadyLinkedCanUnlink = new ArrayList<>(); - //new ArrayList<>(); - if (dvObject instanceof Dataset) { - dataversesAlreadyLinked = ctxt.dsLinking().findLinkingDataverses(dvObject.getId(), searchParam); - for (Dataverse dv : dataversesAlreadyLinked) { - if (ctxt.permissions().hasPermissionsFor(getRequest(), dv, EnumSet.of(Permission.LinkDataset))) { - dataversesAlreadyLinkedCanUnlink.add(dv); - } - return dataversesAlreadyLinkedCanUnlink; - } - } else { - dataversesAlreadyLinked = ctxt.dvLinking().findLinkingDataverses(dvObject.getId(), searchParam); - for (Dataverse dv : dataversesAlreadyLinked) { - if (ctxt.permissions().hasPermissionsFor(getRequest(), dv, EnumSet.of(Permission.LinkDataverse))) { - dataversesAlreadyLinkedCanUnlink.add(dv); - } - } - return dataversesAlreadyLinkedCanUnlink; - } - return null; - } - } -} diff --git a/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java b/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java index e7bcf17d44b..b98f88e386f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java +++ b/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java @@ -312,7 +312,6 @@ public void exportAllFormats(Dataset dataset) throws ExportException { } catch (ServiceConfigurationError serviceError) { throw new ExportException("Service configuration error during export. " + serviceError.getMessage()); } catch (RuntimeException e) { - e.printStackTrace(); logger.log(Level.FINE, e.getMessage(), e); throw new ExportException( "Unknown runtime exception exporting metadata. " + (e.getMessage() == null ? "" : e.getMessage())); diff --git a/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java index fa87926210f..fa56432cc3c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java @@ -39,13 +39,13 @@ public class DatasetExternalCitationsServiceBean implements java.io.Serializable DatasetServiceBean datasetService; //Array of relationship types that are considered to be citations - public static ArrayList inboundRelationships = new ArrayList( + static ArrayList inboundRelationships = new ArrayList( Arrays.asList( "cites", "references", "supplements", "is-supplement-to")); - public static ArrayList outboundRelationships = new ArrayList( + static ArrayList outboundRelationships = new ArrayList( Arrays.asList( "is-cited-by", "is-referenced-by", diff --git a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java index 9ebb346baf8..baf8302437d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java +++ b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java @@ -58,9 +58,6 @@ import edu.harvard.iq.dataverse.util.xml.XmlWriterUtil; import jakarta.enterprise.inject.spi.CDI; import jakarta.json.JsonObject; -import jakarta.json.JsonString; -import jakarta.json.JsonValue; -import jakarta.json.JsonValue.ValueType; public class XmlMetadataTemplate { @@ -624,12 +621,8 @@ private void writeEntityElements(XMLStreamWriter xmlw, String elementName, Strin if (externalIdentifier.isValidIdentifier(orgName)) { isROR = true; JsonObject jo = getExternalVocabularyValue(orgName); - // Some ext. cvv configs store a JsonArray of multiple objects/values. In such cases, we'll leave orgName blank - if (jo != null && jo.containsKey("termName")) { - JsonValue termName = jo.get("termName"); - if (termName.getValueType() == ValueType.STRING) { - orgName = ((JsonString) termName).getString(); - } + if (jo != null) { + orgName = jo.getString("termName"); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java index aec352a615b..64679b05beb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java @@ -28,8 +28,6 @@ import jakarta.ejb.EJB; import jakarta.ejb.Stateless; -import jakarta.ejb.TransactionAttribute; -import jakarta.ejb.TransactionAttributeType; import jakarta.inject.Named; import jakarta.json.Json; import jakarta.json.JsonObjectBuilder; @@ -45,9 +43,6 @@ public class SolrIndexServiceBean { private static final Logger logger = Logger.getLogger(SolrIndexServiceBean.class.getCanonicalName()); - @EJB - private SolrIndexServiceBean self; // Self-injection to allow calling methods in new transactions (from other methods in this bean) - @EJB DvObjectServiceBean dvObjectService; @EJB @@ -322,7 +317,7 @@ private void persistToSolr(Collection docs) throws SolrServer /** * We use the database to determine direct children since there is no - * inheritance. This implementation uses smaller transactions to avoid memory issues. + * inheritance */ public IndexResponse indexPermissionsOnSelfAndChildren(DvObject definitionPoint) { @@ -330,150 +325,134 @@ public IndexResponse indexPermissionsOnSelfAndChildren(DvObject definitionPoint) logger.log(Level.WARNING, "Cannot perform indexPermissionsOnSelfAndChildren with a definitionPoint null"); return null; } - int fileQueryMin = JvmSettings.MIN_FILES_TO_USE_PROXY.lookupOptional(Integer.class).orElse(Integer.MAX_VALUE); - final int[] counter = { 0 }; + int fileQueryMin= JvmSettings.MIN_FILES_TO_USE_PROXY.lookupOptional(Integer.class).orElse(Integer.MAX_VALUE); + List filesToReindexAsBatch = new ArrayList<>(); + /** + * @todo Re-indexing the definition point itself seems to be necessary + * for revoke but not necessarily grant. + */ + + // We don't create a Solr "primary/content" doc for the root dataverse + // so don't create a Solr "permission" doc either. + final int[] counter = {0}; int numObjects = 0; long globalStartTime = System.currentTimeMillis(); - - // Handle the definition point itself in its own transaction - if (definitionPoint instanceof Dataverse dataverse) { - // We don't create a Solr "primary/content" doc for the root dataverse - // so don't create a Solr "permission" doc either. - if (!dataverse.equals(dataverseService.findRootDataverse())) { + if (definitionPoint.isInstanceofDataverse()) { + Dataverse selfDataverse = (Dataverse) definitionPoint; + if (!selfDataverse.equals(dataverseService.findRootDataverse())) { indexPermissionsForOneDvObject(definitionPoint); numObjects++; } - - // Process datasets in batches - List datasetIds = datasetService.findIdsByOwnerId(dataverse.getId()); - int batchSize = 10; // Process 10 datasets per transaction - - for (int i = 0; i < datasetIds.size(); i += batchSize) { - int endIndex = Math.min(i + batchSize, datasetIds.size()); - List batchIds = datasetIds.subList(i, endIndex); - - // Process this batch of datasets in a new transaction - self.indexDatasetBatchInNewTransaction(batchIds, counter, fileQueryMin); - numObjects += batchIds.size(); - - logger.fine("Permission reindexing: Processed batch " + (i/batchSize + 1) + " of " + - (int) Math.ceil(datasetIds.size() / (double) batchSize) + - " dataset batches for dataverse " + dataverse.getId()); - } - } else if (definitionPoint instanceof Dataset dataset) { - // For a single dataset, process it in its own transaction - indexPermissionsForOneDvObject(definitionPoint); - numObjects++; - - // Process the dataset's files in a new transaction - self.indexDatasetFilesInNewTransaction(dataset.getId(), counter, fileQueryMin); - } else { - // For other types (like files), just index in a new transaction - indexPermissionsForOneDvObject(definitionPoint); - numObjects++; - } - - logger.fine("Reindexed permissions for " + counter[0] + " files and " + numObjects + - " datasets/collections in " + (System.currentTimeMillis() - globalStartTime) + " ms"); - - return new IndexResponse("Number of dvObject permissions indexed for " + definitionPoint + ": " + numObjects); - } - - @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) - public void indexDatasetBatchInNewTransaction(List datasetIds, final int[] fileCounter, int fileQueryMin) { - for (Long datasetId : datasetIds) { - Dataset dataset = datasetService.find(datasetId); - if (dataset != null) { + List directChildDatasetsOfDvDefPoint = datasetService.findByOwnerId(selfDataverse.getId()); + for (Dataset dataset : directChildDatasetsOfDvDefPoint) { indexPermissionsForOneDvObject(dataset); + numObjects++; - // Process files for this dataset Map desiredCards = searchPermissionsService.getDesiredCards(dataset); - + long startTime = System.currentTimeMillis(); for (DatasetVersion version : versionsToReIndexPermissionsFor(dataset)) { if (desiredCards.get(version.getVersionState())) { - processDatasetVersionFiles(version, fileCounter, fileQueryMin); + List cachedPerms = searchPermissionsService.findDatasetVersionPerms(version); + String solrIdEnd = getDatasetOrDataFileSolrEnding(version.getVersionState()); + Long versionId = version.getId(); + for (FileMetadata fmd : version.getFileMetadatas()) { + DataFileProxy fileProxy = new DataFileProxy(fmd); + // Since reindexFilesInBatches() re-indexes a file in all versions needed, we should not send a file already in the released version twice + filesToReindexAsBatch.add(fileProxy); + counter[0]++; + if (counter[0] % 100 == 0) { + reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); + filesToReindexAsBatch.clear(); + } + if (counter[0] % 1000 == 0) { + logger.fine("Progress: " + counter[0] + "files permissions reindexed"); + } + } + + // Re-index any remaining files in the datasetversion (so that verionId, etc. remain constants for all files in the batch) + reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); + logger.info("Progress : dataset " + dataset.getId() + " permissions reindexed in " + (System.currentTimeMillis() - startTime) + " ms"); } } } - } - } - - @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) - public void indexDatasetFilesInNewTransaction(Long datasetId, final int[] fileCounter, int fileQueryMin) { - Dataset dataset = datasetService.find(datasetId); - if (dataset != null) { + } else if (definitionPoint.isInstanceofDataset()) { + indexPermissionsForOneDvObject(definitionPoint); + numObjects++; + // index files + Dataset dataset = (Dataset) definitionPoint; Map desiredCards = searchPermissionsService.getDesiredCards(dataset); for (DatasetVersion version : versionsToReIndexPermissionsFor(dataset)) { if (desiredCards.get(version.getVersionState())) { - processDatasetVersionFiles(version, fileCounter, fileQueryMin); - } - } - } - } - - private void processDatasetVersionFiles(DatasetVersion version, - final int[] fileCounter, int fileQueryMin) { - List cachedPerms = searchPermissionsService.findDatasetVersionPerms(version); - String solrIdEnd = getDatasetOrDataFileSolrEnding(version.getVersionState()); - Long versionId = version.getId(); - List filesToReindexAsBatch = new ArrayList<>(); - - // Process files in batches of 100 - int batchSize = 100; - - if (version.getFileMetadatas().size() > fileQueryMin) { - // For large datasets, use a more efficient SQL query - Stream fileStream = getDataFileInfoForPermissionIndexing(version.getId()); - - // Process files in batches to avoid memory issues - fileStream.forEach(fileInfo -> { - filesToReindexAsBatch.add(fileInfo); - fileCounter[0]++; - - if (filesToReindexAsBatch.size() >= batchSize) { + List cachedPerms = searchPermissionsService.findDatasetVersionPerms(version); + String solrIdEnd = getDatasetOrDataFileSolrEnding(version.getVersionState()); + Long versionId = version.getId(); + if (version.getFileMetadatas().size() > fileQueryMin) { + // For large datasets, use a more efficient SQL query instead of loading all file metadata objects + getDataFileInfoForPermissionIndexing(version.getId()).forEach(fileInfo -> { + filesToReindexAsBatch.add(fileInfo); + counter[0]++; + + if (counter[0] % 100 == 0) { + long startTime = System.currentTimeMillis(); + reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); + filesToReindexAsBatch.clear(); + logger.fine("Progress: 100 file permissions at " + counter[0] + " files reindexed in " + (System.currentTimeMillis() - startTime) + " ms"); + } + }); + } else { + version.getFileMetadatas().stream() + .forEach(fmd -> { + DataFileProxy fileProxy = new DataFileProxy(fmd); + filesToReindexAsBatch.add(fileProxy); + counter[0]++; + if (counter[0] % 100 == 0) { + long startTime = System.currentTimeMillis(); + reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); + filesToReindexAsBatch.clear(); + logger.fine("Progress: 100 file permissions at " + counter[0] + "files reindexed in " + (System.currentTimeMillis() - startTime) + " ms"); + } + }); + } + // Re-index any remaining files in the dataset version (versionId, etc. remain constants for all files in the batch) reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); filesToReindexAsBatch.clear(); } - }); - } else { - // For smaller datasets, process files directly - for (FileMetadata fmd : version.getFileMetadatas()) { - DataFileProxy fileProxy = new DataFileProxy(fmd); - filesToReindexAsBatch.add(fileProxy); - fileCounter[0]++; - if (filesToReindexAsBatch.size() >= batchSize) { - reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); - filesToReindexAsBatch.clear(); - } } + } else { + indexPermissionsForOneDvObject(definitionPoint); + numObjects++; } - // Process any remaining files - if (!filesToReindexAsBatch.isEmpty()) { - reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); - } + /** + * @todo Error handling? What to do with response? + * + * @todo Should update timestamps, probably, even thought these are files, see + * https://github.com/IQSS/dataverse/issues/2421 + */ + logger.fine("Reindexed permissions for " + counter[0] + " files and " + numObjects + "datasets/collections in " + (System.currentTimeMillis() - globalStartTime) + " ms"); + return new IndexResponse("Number of dvObject permissions indexed for " + definitionPoint + + ": " + numObjects); } - private void reindexFilesInBatches(List filesToReindexAsBatch, List cachedPerms, Long versionId, String solrIdEnd) { + private String reindexFilesInBatches(List filesToReindexAsBatch, List cachedPerms, Long versionId, String solrIdEnd) { List docs = new ArrayList<>(); try { // Assume all files have the same owner if (filesToReindexAsBatch.isEmpty()) { - logger.warning("reindexFilesInBatches called incorrectly with an empty file list"); + return "No files to reindex"; } - for (DataFileProxy file : filesToReindexAsBatch) { + for (DataFileProxy file : filesToReindexAsBatch) { - DvObjectSolrDoc fileSolrDoc = constructDatafileSolrDoc(file, cachedPerms, versionId, solrIdEnd); - SolrInputDocument solrDoc = SearchUtil.createSolrDoc(fileSolrDoc); - docs.add(solrDoc); - } + DvObjectSolrDoc fileSolrDoc = constructDatafileSolrDoc(file, cachedPerms, versionId, solrIdEnd); + SolrInputDocument solrDoc = SearchUtil.createSolrDoc(fileSolrDoc); + docs.add(solrDoc); + } persistToSolr(docs); - logger.fine("Indexed " + filesToReindexAsBatch.size() + " files across " + docs.size() + " Solr documents"); + return " " + filesToReindexAsBatch.size() + " files indexed across " + docs.size() + " Solr documents "; } catch (SolrServerException | IOException ex) { - logger.log(Level.WARNING, "Failed to reindex " + filesToReindexAsBatch.size() + - " files across " + docs.size() + " Solr documents", ex); + return " tried to reindex " + filesToReindexAsBatch.size() + " files indexed across " + docs.size() + " Solr documents but caught exception: " + ex; } } diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java index 87123801a3e..53dff244ae1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java @@ -93,10 +93,6 @@ public enum JvmSettings { API_BLOCKED_ENDPOINTS(SCOPE_API_BLOCKED, "endpoints"), API_BLOCKED_POLICY(SCOPE_API_BLOCKED, "policy"), API_BLOCKED_KEY(SCOPE_API_BLOCKED, "key"), - // API: MDC Citation updates - SCOPE_API_MDC(SCOPE_API, "mdc"), - API_MDC_UPDATE_MIN_DELAY_MS(SCOPE_API_MDC, "min-delay-ms"), - // SIGNPOSTING SETTINGS SCOPE_SIGNPOSTING(PREFIX, "signposting"), diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index 8b0ea201aa3..bbc834e0cc4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -40,7 +40,6 @@ import edu.harvard.iq.dataverse.workflow.Workflow; import edu.harvard.iq.dataverse.workflow.step.WorkflowStepData; -import java.io.IOException; import java.util.*; import jakarta.json.Json; import jakarta.json.JsonArrayBuilder; @@ -291,35 +290,22 @@ public static JsonObjectBuilder json(Workflow wf){ return bld; } - - public static JsonObjectBuilder json(Dataverse dv, boolean minimal) { - if (!minimal){ - return json(dv, false, false, false, null); - } else { - return json(dv, false, false, true, null); - } - } public static JsonObjectBuilder json(Dataverse dv) { - return json(dv, false, false, false, null); + return json(dv, false, false, null); } //TODO: Once we upgrade to Java EE 8 we can remove objects from the builder, and this email removal can be done in a better place. - public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean returnOwners, Boolean minimal, Long childCount) { + public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean returnOwners, Long childCount) { JsonObjectBuilder bld = jsonObjectBuilder() .add("id", dv.getId()) .add("alias", dv.getAlias()) - .add("name", dv.getName()); - //minimal refers to only returning the id alias and name for - //used in selecting collections available for linking - if (minimal) { - return bld; - } - bld.add("affiliation", dv.getAffiliation()); - if (!hideEmail) { + .add("name", dv.getName()) + .add("affiliation", dv.getAffiliation()); + if(!hideEmail) { bld.add("dataverseContacts", JsonPrinter.json(dv.getDataverseContacts())); } - if (returnOwners) { + if (returnOwners){ bld.add("isPartOf", getOwnersFromDvObject(dv)); } bld.add("permissionRoot", dv.isPermissionRoot()) @@ -336,8 +322,8 @@ public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean re if (dv.getDataverseTheme() != null) { bld.add("theme", JsonPrinter.json(dv.getDataverseTheme())); } - if (dv.getStorageDriverId() != null) { - bld.add("storageDriverLabel", DataAccess.getStorageDriverLabelFor(dv.getStorageDriverId())); + if(dv.getStorageDriverId() != null) { + bld.add("storageDriverLabel", DataAccess.getStorageDriverLabelFor(dv.getStorageDriverId())); } if (dv.getFilePIDsEnabled() != null) { bld.add("filePIDsEnabled", dv.getFilePIDsEnabled()); @@ -346,7 +332,7 @@ public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean re bld.add("isReleased", dv.isReleased()); List inputLevels = dv.getDataverseFieldTypeInputLevels(); - if (!inputLevels.isEmpty()) { + if(!inputLevels.isEmpty()) { bld.add("inputLevels", JsonPrinter.jsonDataverseFieldTypeInputLevels(inputLevels)); } @@ -354,6 +340,7 @@ public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean re bld.add("childCount", childCount); } addDatasetFileCountLimit(dv, bld); + return bld; } @@ -1666,23 +1653,6 @@ public static JsonArrayBuilder jsonTemplateInstructions(Map temp return jsonArrayBuilder; } - public static JsonObjectBuilder jsonStorageDriver(String storageDriverId, Dataset dataset) { - JsonObjectBuilder jsonObjectBuilder = new NullSafeJsonBuilder(); - jsonObjectBuilder.add("name", storageDriverId); - jsonObjectBuilder.add("type", DataAccess.getDriverType(storageDriverId)); - jsonObjectBuilder.add("label", DataAccess.getStorageDriverLabelFor(storageDriverId)); - if (dataset != null) { - jsonObjectBuilder.add("directUpload", DataAccess.uploadToDatasetAllowed(dataset, storageDriverId)); - try { - jsonObjectBuilder.add("directDownload", DataAccess.getStorageIO(dataset).downloadRedirectEnabled()); - } catch (IOException ex) { - logger.fine("Failed to get Storage IO for dataset " + ex.getMessage()); - } - } - - return jsonObjectBuilder; - } - public static JsonArrayBuilder json(List notifications, AuthenticatedUser authenticatedUser, boolean inAppNotificationFormat) { JsonArrayBuilder notificationsArray = Json.createArrayBuilder(); diff --git a/src/main/webapp/WEB-INF/glassfish-resources.xml b/src/main/webapp/WEB-INF/glassfish-resources.xml index 74af3be42ce..3fbbf4c3586 100644 --- a/src/main/webapp/WEB-INF/glassfish-resources.xml +++ b/src/main/webapp/WEB-INF/glassfish-resources.xml @@ -11,14 +11,4 @@ - - - - - - - - \ No newline at end of file diff --git a/src/main/webapp/dataset.xhtml b/src/main/webapp/dataset.xhtml index 7729b6da442..dbc1fb3aeae 100644 --- a/src/main/webapp/dataset.xhtml +++ b/src/main/webapp/dataset.xhtml @@ -525,7 +525,7 @@ #{bundle['dataset.curationStatusMenu']}
    - +
  • #{bundle['dataset.changestatus']}
      diff --git a/src/main/webapp/file.xhtml b/src/main/webapp/file.xhtml index 2292ebf4c45..8a5c2709e5d 100644 --- a/src/main/webapp/file.xhtml +++ b/src/main/webapp/file.xhtml @@ -374,7 +374,6 @@ - @@ -682,7 +681,6 @@ - diff --git a/src/main/webapp/guestbook-terms-popup-fragment.xhtml b/src/main/webapp/guestbook-terms-popup-fragment.xhtml index a17145a43ce..7d157fcb72c 100644 --- a/src/main/webapp/guestbook-terms-popup-fragment.xhtml +++ b/src/main/webapp/guestbook-terms-popup-fragment.xhtml @@ -142,7 +142,7 @@ value="#{MarkupChecker:sanitizeBasicHTML(workingVersion.termsOfUseAndAccess.termsOfAccess)}" escape="false" /> - +
      diff --git a/src/main/webapp/resources/css/structure.css b/src/main/webapp/resources/css/structure.css index 39e96390dc7..4d83ddd489d 100644 --- a/src/main/webapp/resources/css/structure.css +++ b/src/main/webapp/resources/css/structure.css @@ -7,9 +7,6 @@ html { body { /* Sticky footer Margin bottom by footer height */ margin-bottom:160px; - /* #11824 */ - word-break: normal; - overflow-wrap: anywhere; } body.widget-view {margin-bottom:44px;} body .ui-widget {font-size: inherit;} diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java index 16552869a80..b273502e6ed 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java @@ -3875,8 +3875,6 @@ public void testCurationStatusAPIs() { createDataverseResponse.prettyPrint(); String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse); - UtilIT.publishDataverseViaNativeApi(dataverseAlias, apiToken).then().assertThat().statusCode(OK.getStatusCode()); - Response setCurationLabelSets = UtilIT.setSetting(SettingsServiceBean.Key.AllowedCurationLabels, "{\"StandardProcess\":[\"Author contacted\", \"Privacy Review\", \"Awaiting paper publication\", \"Final Approval\"],\"AlternateProcess\":[\"State 1\",\"State 2\",\"State 3\"]}"); setCurationLabelSets.then().assertThat() .statusCode(OK.getStatusCode()); @@ -3887,7 +3885,7 @@ public void testCurationStatusAPIs() { Response setDataverseCurationLabelSetResponse = UtilIT.setDataverseCurationLabelSet(dataverseAlias, apiToken, "AlternateProcess"); setDataverseCurationLabelSetResponse.then().assertThat().statusCode(FORBIDDEN.getStatusCode()); - Response makeSuperUser = UtilIT.setSuperuserStatus(username, true); + Response makeSuperUser = UtilIT.makeSuperUser(username); assertEquals(200, makeSuperUser.getStatusCode()); //Non-existent option @@ -3946,66 +3944,8 @@ public void testCurationStatusAPIs() { Response setInvalidStatus = UtilIT.setDatasetCurationLabel(datasetId, apiToken, "Invalid Status"); setInvalidStatus.then().assertThat().statusCode(BAD_REQUEST.getStatusCode()); - // Publish the dataset - UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken).then().assertThat() - .statusCode(OK.getStatusCode()); - - // Verify that the current curation label is now empty after publishing - Response getStatusAfterPublish = UtilIT.getDatasetCurationStatus(datasetId, apiToken, false); - getStatusAfterPublish.then().assertThat().statusCode(OK.getStatusCode()); - JsonObject statusAfterPublish = Json.createReader(new StringReader(getStatusAfterPublish.body().asString())).readObject(); - JsonObject dataObject = statusAfterPublish.getJsonObject("data"); - assertFalse(dataObject.containsKey("label"), "Curation label should be empty after publishing"); - - //Cause a new draft version - String jsonLDTerms = "{\"https://dataverse.org/schema/core#fileTermsOfAccess\":{\"https://dataverse.org/schema/core#dataAccessPlace\":\"Somewhere\"}}"; - Response updateTerms = UtilIT.updateDatasetJsonLDMetadata(datasetId, apiToken, jsonLDTerms, true); - updateTerms.then().assertThat() - .statusCode(OK.getStatusCode()); - - // Add a new valid curation label - Response setNewStatus = UtilIT.setDatasetCurationLabel(datasetId, apiToken, "State 2"); - setNewStatus.then().assertThat().statusCode(OK.getStatusCode()); - - // Verify the label was set - Response getStatusAfterSet = UtilIT.getDatasetCurationStatus(datasetId, apiToken, false); - getStatusAfterSet.then().assertThat().statusCode(OK.getStatusCode()); - JsonObject statusAfterSet = Json.createReader(new StringReader(getStatusAfterSet.body().asString())).readObject(); - JsonObject dataInSecondDraft = statusAfterSet.getJsonObject("data"); - assertEquals("State 2", dataInSecondDraft.getString("label"), "Curation label should be set to State 2"); - - // Publish the dataset again using updatecurrent as superuser - Response updateCurrentResponse = UtilIT.publishDatasetViaNativeApi(datasetId, "updatecurrent", apiToken); - updateCurrentResponse.then().assertThat().statusCode(OK.getStatusCode()); - - // Verify that the current curation label is now empty after updatecurrent - Response getStatusAfterUpdateCurrent = UtilIT.getDatasetCurationStatus(datasetId, apiToken, false); - getStatusAfterUpdateCurrent.then().assertThat().statusCode(OK.getStatusCode()); - JsonObject statusAfterUpdateCurrent = Json.createReader(new StringReader(getStatusAfterUpdateCurrent.body().asString())).readObject(); - JsonObject dataAfterUpdateCurrent = statusAfterUpdateCurrent.getJsonObject("data"); - assertFalse(dataAfterUpdateCurrent.containsKey("label"), "Curation label should be empty after updatecurrent"); - - // Verify that the history contains the previously added label - Response getHistoryAfterUpdateCurrent = UtilIT.getDatasetCurationStatus(datasetId, apiToken, true); - getHistoryAfterUpdateCurrent.then().assertThat().statusCode(OK.getStatusCode()); - - // Extract the data array from the response - JsonObject responseObj = Json.createReader(new StringReader(getHistoryAfterUpdateCurrent.body().asString())).readObject(); - JsonArray historyAfterUpdateCurrent = responseObj.getJsonArray("data"); - - // Verify history contains the State 2 label - boolean foundState2 = false; - for (int i = 0; i < historyAfterUpdateCurrent.size(); i++) { - JsonObject entry = historyAfterUpdateCurrent.getJsonObject(i); - if (entry.containsKey("label") && "State 2".equals(entry.getString("label"))) { - foundState2 = true; - break; - } - } - assertTrue(foundState2, "History should contain the State 2 label after updatecurrent"); - // Clean up - Response deleteDatasetResponse = UtilIT.destroyDataset(datasetId, apiToken); + Response deleteDatasetResponse = UtilIT.deleteDatasetViaNativeApi(datasetId, apiToken); assertEquals(200, deleteDatasetResponse.getStatusCode()); Response deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAlias, apiToken); @@ -6118,60 +6058,6 @@ public void testGetGlobusUploadParameters() { GlobusOverlayAccessIOTest.tearDown(); } - @Test - public void testSetGetDatasetStorageDriver() { - Response createUser = UtilIT.createRandomUser(); - createUser.then().assertThat().statusCode(OK.getStatusCode()); - String apiToken = UtilIT.getApiTokenFromResponse(createUser); - String username = UtilIT.getUsernameFromResponse(createUser); - Response makeSuperUser = UtilIT.makeSuperUser(username); - assertEquals(200, makeSuperUser.getStatusCode()); - Response createDataverse = UtilIT.createRandomDataverse(apiToken); - String dataverseAlias = UtilIT.getAliasFromResponse(createDataverse); - Response createDataset = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); - Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); - - Response storageDrivers = UtilIT.listStorageDrivers(apiToken); - storageDrivers.prettyPrint(); - JsonObject data = JsonUtil.getJsonObject(storageDrivers.getBody().asString()); - String first = data.getJsonObject("data").keySet().iterator().next(); - String name = data.getJsonObject("data").getString(first); - - Response setDriver = UtilIT.setDatasetStorageDriver(datasetId, first, apiToken); - setDriver.prettyPrint(); - assertEquals(200, setDriver.getStatusCode()); - Response getDriver = UtilIT.getDatasetStorageDriver(datasetId, apiToken); - getDriver.prettyPrint(); - assertEquals(200, getDriver.getStatusCode()); - getDriver.then().assertThat() - .body("data.name", CoreMatchers.equalTo(name)) - .body("data.type", CoreMatchers.notNullValue()) - .body("data.label", CoreMatchers.notNullValue()) - .body("data.directUpload", CoreMatchers.notNullValue()) - .body("data.directDownload", CoreMatchers.notNullValue()) - .statusCode(OK.getStatusCode()); - - // Test dataset under root with default storage driver - Response getStorageDriverResponse = UtilIT.getStorageDriver("root", apiToken, Boolean.TRUE); - getStorageDriverResponse.prettyPrint(); - data = JsonUtil.getJsonObject(getStorageDriverResponse.getBody().asString()); - name = data.getJsonObject("data").getString("name"); - String type = data.getJsonObject("data").getString("type"); - String label = data.getJsonObject("data").getString("label"); - createDataset = UtilIT.createRandomDatasetViaNativeApi("root", apiToken); - datasetId = UtilIT.getDatasetIdFromResponse(createDataset); - getDriver = UtilIT.getDatasetStorageDriver(datasetId, apiToken); - getDriver.prettyPrint(); - assertEquals(200, getDriver.getStatusCode()); - getDriver.then().assertThat() - .body("data.name", CoreMatchers.equalTo(name)) - .body("data.type", CoreMatchers.equalTo(type)) - .body("data.label", CoreMatchers.equalTo(label)) - .body("data.directUpload", CoreMatchers.notNullValue()) - .body("data.directDownload", CoreMatchers.notNullValue()) - .statusCode(OK.getStatusCode()); - } - @Test public void testGetCanDownloadAtLeastOneFile() { Response createUserResponse = UtilIT.createRandomUser(); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java index eb247ce6c7f..4f9fbae7ffc 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java @@ -1,6 +1,5 @@ package edu.harvard.iq.dataverse.api; -import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.util.json.JsonParseException; import edu.harvard.iq.dataverse.util.json.JsonParser; import edu.harvard.iq.dataverse.util.json.JsonUtil; @@ -46,7 +45,6 @@ import io.restassured.path.json.JsonPath; import org.hamcrest.CoreMatchers; import org.hamcrest.Matchers; -import static org.hamcrest.Matchers.greaterThan; public class DataversesIT { @@ -679,235 +677,6 @@ public void testImportDDI() throws IOException, InterruptedException { Response deleteUserResponse = UtilIT.deleteUser(username); assertEquals(200, deleteUserResponse.getStatusCode()); } - - @Test - public void testGetLinkableDataverses(){ - Response createUser = UtilIT.createRandomUser(); - String apiToken = UtilIT.getApiTokenFromResponse(createUser); - String username = UtilIT.getUsernameFromResponse(createUser); - - Response createDataverseResponse = UtilIT.createRandomDataverse(apiToken); - String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse); - - //Create dataverse for linking - Response createDataverseResponseForLinking = UtilIT.createRandomDataverse(apiToken); - String dataverseAliasForLinking = UtilIT.getAliasFromResponse(createDataverseResponseForLinking); - - - Response publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAlias, apiToken); - assertEquals(200, publishDataverse.getStatusCode()); - - publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAliasForLinking, apiToken); - assertEquals(200, publishDataverse.getStatusCode()); - - Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); - createDatasetResponse.prettyPrint(); - String datasetPersistentId = UtilIT.getDatasetPersistentIdFromResponse(createDatasetResponse); - - Integer datasetId = UtilIT.getDatasetIdFromResponse(createDatasetResponse); - UtilIT.publishDatasetViaNativeApi(datasetPersistentId, "major", apiToken); - - Response getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasForLinking); - getLinkableDataverses.prettyPrint(); - getLinkableDataverses.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[0].alias", equalTo(dataverseAliasForLinking)); - - Response getLinkableDataversesForDataverse = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, dataverseAliasForLinking); - getLinkableDataversesForDataverse.prettyPrint(); - getLinkableDataverses.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[0].alias", equalTo(dataverseAliasForLinking)); - - //Should be able to get based on a partial alias... - // Partial must include the first part of the name - String searchTerm = dataverseAliasForLinking.substring(0, 7); - - Response getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, searchTerm); - getLinkableDataversesForDataversePartial.prettyPrint(); - getLinkableDataversesForDataversePartial.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[0].alias", equalTo(dataverseAliasForLinking)); - - //if I give a blank search term i should get the one that I have perms on - Response getLinkableDataversesForDataverseBlank = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, ""); - getLinkableDataversesForDataverseBlank.prettyPrint(); - getLinkableDataversesForDataverseBlank.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[0].alias", equalTo(dataverseAliasForLinking)); - - - //Try with empty string search term - searchTerm = ""; - getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, searchTerm); - getLinkableDataversesForDataversePartial.prettyPrint(); - getLinkableDataversesForDataversePartial.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[0].alias", equalTo(dataverseAliasForLinking)); - - //Should get same result if search term is null - getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, null); - getLinkableDataversesForDataversePartial.prettyPrint(); - getLinkableDataversesForDataversePartial.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[0].alias", equalTo(dataverseAliasForLinking)); - - // Create a child of the linking DV - Response createLevel1a = UtilIT.createSubDataverse(UtilIT.getRandomDvAlias() + "-level1a", null, apiToken, dataverseAliasForLinking); - createLevel1a.prettyPrint(); - String childLinking = UtilIT.getAliasFromResponse(createLevel1a); - - //Try to link the child - should not link to parent - should link to "uncle" - getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", childLinking, apiToken, searchTerm); - getLinkableDataversesForDataversePartial.prettyPrint(); - getLinkableDataversesForDataversePartial.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[0].alias", equalTo(dataverseAlias)); - - //Try with bad target alias - searchTerm = ""; - Response getLinkableDataversesForDataverseBadId = UtilIT.getLinkableDataverses("dataverse", "junque@#", apiToken, searchTerm); - getLinkableDataversesForDataverseBadId.prettyPrint(); - getLinkableDataversesForDataverseBadId.then().assertThat() - .statusCode(NOT_FOUND.getStatusCode()); - - // create new user and dataverse - the new dataverse should not be available to the first user for linking... - Response createUserTwo = UtilIT.createRandomUser(); - String apiTokenTwo = UtilIT.getApiTokenFromResponse(createUserTwo); - String usernameTwo = UtilIT.getUsernameFromResponse(createUserTwo); - - //Create dataverse that should be unavailable for linking - Response createDataverseResponseUnavailableForLinking = UtilIT.createRandomDataverse(apiTokenTwo); - createDataverseResponseUnavailableForLinking.prettyPrint(); - String dataverseAliasUnavailableForLinking = UtilIT.getAliasFromResponse(createDataverseResponseUnavailableForLinking); - publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAliasUnavailableForLinking, apiTokenTwo); - publishDataverse.prettyPrint(); - assertEquals(200, publishDataverse.getStatusCode()); - - //user 3 will not have permissions - Response createUserThree = UtilIT.createRandomUser(); - String apiTokenThree = UtilIT.getApiTokenFromResponse(createUserThree); - String usernameThree = UtilIT.getUsernameFromResponse(createUserThree); - - Response getUnavailableForDatasetBadSearchTerm = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, "xxQQQ-Batman"); - getUnavailableForDatasetBadSearchTerm.prettyPrint(); - getUnavailableForDatasetBadSearchTerm.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(0)); - - Response getUnavailableForDataset = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasUnavailableForLinking); - getUnavailableForDataset.prettyPrint(); - getUnavailableForDataset.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(0)); - - Response getGuestUnavailableForDataset = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, "", dataverseAliasUnavailableForLinking); - getGuestUnavailableForDataset.prettyPrint(); - getGuestUnavailableForDataset.then().assertThat() - .statusCode(FORBIDDEN.getStatusCode()); - - Response getUnavailableForDataverse = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, dataverseAliasUnavailableForLinking); - getUnavailableForDataverse.prettyPrint(); - getUnavailableForDataverse.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(0)); - - Response getNoPermsOnAnyCollection = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiTokenThree, ""); - getNoPermsOnAnyCollection.prettyPrint(); - getNoPermsOnAnyCollection.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(0)); - - //now link a dataset and see that it's unavailable in the future - - Response makeSuperUser = UtilIT.setSuperuserStatus(username, Boolean.TRUE); - - Response linkDataset = UtilIT.linkDataset(datasetPersistentId, dataverseAliasForLinking, apiToken); - linkDataset.prettyPrint(); - linkDataset.then().assertThat() - .statusCode(OK.getStatusCode()); - - //set user api back to non-super user so perms are limited - UtilIT.setSuperuserStatus(username, Boolean.FALSE); - - //should get an empty list because dataset is already linked - getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasForLinking); - getLinkableDataverses.prettyPrint(); - getLinkableDataverses.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(0)); - - //if you ask for already linked you should get one - getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasForLinking, true); - - getLinkableDataverses.prettyPrint(); - getLinkableDataverses.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(1)); - - //if you ask for already linked you should get one unless you don't have perms - getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiTokenThree, dataverseAliasForLinking, true); - - getLinkableDataverses.prettyPrint(); - getLinkableDataverses.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(0)); - - //if you ask for already linked you should get one unless there's a bad search terms - getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, "QQQBatmanSymbol", true); - - getLinkableDataverses.prettyPrint(); - getLinkableDataverses.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(0)); - - //set user api back to super user for cleanup - UtilIT.setSuperuserStatus(username, Boolean.TRUE); - - //Create dataverse that should be available for super user - Response createDataverseResponseAvailableForLinkingBySuperUser = UtilIT.createRandomDataverse(apiTokenTwo); - createDataverseResponseAvailableForLinkingBySuperUser.prettyPrint(); - String dataverseAliasAvailableForLinkingBySuperUser = UtilIT.getAliasFromResponse(createDataverseResponseAvailableForLinkingBySuperUser); - publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAliasAvailableForLinkingBySuperUser, apiToken); - publishDataverse.prettyPrint(); - assertEquals(200, publishDataverse.getStatusCode()); - - //First see that the super user gets dataverses to link to - getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasAvailableForLinkingBySuperUser); - getLinkableDataverses.prettyPrint(); - getLinkableDataverses.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.size()", equalTo(1)); - - // Clean up - Response destroyDatasetResponse = UtilIT.destroyDataset(datasetId, apiToken); - assertEquals(200, destroyDatasetResponse.getStatusCode()); - - Response deleteDataverseResponseSuper = UtilIT.deleteDataverse(dataverseAliasAvailableForLinkingBySuperUser, apiToken); - assertEquals(200, deleteDataverseResponseSuper.getStatusCode()); - - Response deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAlias, apiToken); - assertEquals(200, deleteDataverseResponse.getStatusCode()); - - deleteDataverseResponse = UtilIT.deleteDataverse(childLinking, apiToken); - assertEquals(200, deleteDataverseResponse.getStatusCode()); - - deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAliasForLinking, apiToken); - assertEquals(200, deleteDataverseResponse.getStatusCode()); - - deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAliasUnavailableForLinking, apiToken); - assertEquals(200, deleteDataverseResponse.getStatusCode()); - - Response deleteUserResponse = UtilIT.deleteUser(usernameTwo); - assertEquals(200, deleteUserResponse.getStatusCode()); - - deleteUserResponse = UtilIT.deleteUser(usernameThree); - assertEquals(200, deleteUserResponse.getStatusCode()); - - deleteUserResponse = UtilIT.deleteUser(username); - assertEquals(200, deleteUserResponse.getStatusCode()); - - } @Test public void testImport() throws IOException, InterruptedException { @@ -1486,77 +1255,6 @@ public void testUpdateInputLevels() { updateDataverseInputLevelsResponse.then().assertThat() .body("message", equalTo("Error while updating dataverse input levels: Input level list cannot be null or empty")) .statusCode(INTERNAL_SERVER_ERROR.getStatusCode()); - - //Add new types and see that previously changed ones remain as before... #11387 - testInputLevelNames = new String[]{"subtitle", "relatedMaterial"}; - - testRequiredInputLevels = new boolean[] {false, false}; - testIncludedInputLevels = new boolean[] {true, true}; - boolean [] testDisplayOnCreate = new boolean[] {true, false}; - updateDataverseInputLevelsResponse = UtilIT.updateDataverseInputLevels(dataverseAlias, testInputLevelNames, testRequiredInputLevels, testIncludedInputLevels, testDisplayOnCreate, apiToken); - updateDataverseInputLevelsResponse.prettyPrint(); - int subtitleInputLevelIndex = -1; - int relatedMaterialInputLevelIndex = -1; - int i = 0; - - while (updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)) != null){ - actualInputLevelName = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)).toString(); - if (actualInputLevelName.equals("subtitle")){ - subtitleInputLevelIndex = i; - } - if (actualInputLevelName.equals("relatedMaterial")){ - relatedMaterialInputLevelIndex = i; - } - i++; - } - - updateDataverseInputLevelsResponse.then().assertThat() - .body(String.format("data.inputLevels[%d].include", subtitleInputLevelIndex), equalTo(true)) - .body(String.format("data.inputLevels[%d].required", subtitleInputLevelIndex), equalTo(false)) - .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleInputLevelIndex), equalTo(true)) - .body(String.format("data.inputLevels[%d].include", relatedMaterialInputLevelIndex), equalTo(true)) - .body(String.format("data.inputLevels[%d].required", relatedMaterialInputLevelIndex), equalTo(false)) - .body(String.format("data.inputLevels[%d].displayOnCreate", relatedMaterialInputLevelIndex), equalTo(false)) - .statusCode(OK.getStatusCode()); - - actualFieldTypeName1 = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", subtitleInputLevelIndex)); - actualFieldTypeName2 = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", relatedMaterialInputLevelIndex)); - assertNotEquals(actualFieldTypeName1, actualFieldTypeName2); - assertThat(testInputLevelNames, hasItemInArray(actualFieldTypeName1)); - assertThat(testInputLevelNames, hasItemInArray(actualFieldTypeName2)); - - - testInputLevelNames = new String[]{"subtitle", "otherReferences"}; - testRequiredInputLevels = new boolean[] {false, false}; - testIncludedInputLevels = new boolean[] {true, true}; - testDisplayOnCreate = new boolean[] {false, true}; - - updateDataverseInputLevelsResponse = UtilIT.updateDataverseInputLevels(dataverseAlias, testInputLevelNames, testRequiredInputLevels, testIncludedInputLevels, testDisplayOnCreate, apiToken); - updateDataverseInputLevelsResponse.prettyPrint(); - - subtitleInputLevelIndex = 0; - i = 0; - - while (updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)) != null) { - actualInputLevelName = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)).toString(); - if (actualInputLevelName.equals("subtitle")) { - subtitleInputLevelIndex = i; - } - i++; - } - - //make sure subtitle got changed to false - updateDataverseInputLevelsResponse.then().assertThat() - .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleInputLevelIndex), equalTo(false)) - .statusCode(OK.getStatusCode()); - - //make superuser for cleanup - String username = UtilIT.getUsernameFromResponse(createUserResponse); - UtilIT.setSuperuserStatus(username, Boolean.TRUE); - Response deleteDataverse1Response = UtilIT.deleteDataverse(dataverseAlias, apiToken); - deleteDataverse1Response.prettyPrint(); - assertEquals(200, deleteDataverse1Response.getStatusCode()); - } @Test @@ -2558,17 +2256,12 @@ public void testUpdateInputLevelDisplayOnCreateOverride() { .body("data.inputLevels[0].displayOnCreate", equalTo(true)) .body("data.inputLevels[0].datasetFieldTypeName", equalTo("notesText")); - updateResponse = UtilIT.updateDataverseInputLevelDisplayOnCreate( dataverseAlias, "subtitle", true, apiToken); - - String actualInputLevelName = updateResponse.then().extract().path("data.inputLevels[0].datasetFieldTypeName"); - int subtitleInputLevelIndex = actualInputLevelName.equals("subtitle") ? 0 : 1; - updateResponse.prettyPrint(); updateResponse.then().assertThat() .statusCode(OK.getStatusCode()) - .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleInputLevelIndex), equalTo(true)) - .body(String.format("data.inputLevels[%d].datasetFieldTypeName", subtitleInputLevelIndex), equalTo("subtitle")); + .body("data.inputLevels[0].displayOnCreate", equalTo(true)) + .body("data.inputLevels[0].datasetFieldTypeName", equalTo("subtitle")); listMetadataBlocksResponse = UtilIT.listMetadataBlocks(dataverseAlias, true, true, apiToken); listMetadataBlocksResponse.prettyPrint(); @@ -2581,51 +2274,28 @@ public void testUpdateInputLevelDisplayOnCreateOverride() { .body("data[0].displayName", equalTo("Citation Metadata")) .body("data.size()", equalTo(expectedOnlyDisplayedOnCreateNumberOfMetadataBlocks)) .body("data[0].fields.author.childFields.size()", is(4)); - - updateResponse = UtilIT.updateDataverseInputLevelDisplayOnCreate(dataverseAlias, "subtitle", false, apiToken); - actualInputLevelName = updateResponse.then().extract().path("data.inputLevels[0].datasetFieldTypeName"); - int subtitleIndex = actualInputLevelName.equals("subtitle") ? 0 : 1; + + updateResponse = UtilIT.updateDataverseInputLevelDisplayOnCreate( + dataverseAlias, "subtitle", false, apiToken); updateResponse.then().assertThat() - .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleIndex), equalTo(false)) - .body(String.format("data.inputLevels[%d].datasetFieldTypeName", subtitleIndex), equalTo("subtitle")) - .statusCode(OK.getStatusCode()); - + .statusCode(OK.getStatusCode()) + .body("data.inputLevels[0].displayOnCreate", equalTo(false)) + .body("data.inputLevels[0].datasetFieldTypeName", equalTo("subtitle")); + } @Test - public void testCreateAndGetTemplates() throws JsonParseException { + public void testCreateAndGetTemplates() { Response createUserResponse = UtilIT.createRandomUser(); String apiToken = UtilIT.getApiTokenFromResponse(createUserResponse); Response createSecondUserResponse = UtilIT.createRandomUser(); String secondApiToken = UtilIT.getApiTokenFromResponse(createSecondUserResponse); - - /* - We need to make this a non-inherited metadatablocks so the get template will only get templates from current dv - */ - + Response createDataverseResponse = UtilIT.createRandomDataverse(apiToken); createDataverseResponse.then().assertThat().statusCode(CREATED.getStatusCode()); String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse); - String newName = "New Test Dataverse Name"; - String newAffiliation = "New Test Dataverse Affiliation"; - String newDataverseType = Dataverse.DataverseType.TEACHING_COURSES.toString(); - String[] newContactEmails = new String[]{"new_email@dataverse.com"}; - String[] newInputLevelNames = new String[]{"geographicCoverage"}; - String[] newFacetIds = new String[]{"contributorName"}; - String[] newMetadataBlockNames = new String[]{"citation", "geospatial", "biomedical"}; - - //Giving the new Dataverse updated metadatablocks so that it will not inherit templates - Response updateDataverseResponse = UtilIT.updateDataverse( - dataverseAlias, dataverseAlias, newName, newAffiliation, newDataverseType, newContactEmails, newInputLevelNames, - null, newMetadataBlockNames, apiToken, - Boolean.FALSE, Boolean.FALSE, null - ); - - updateDataverseResponse.then().assertThat() - .statusCode(OK.getStatusCode()); - // Create a template String jsonString = """ @@ -2663,7 +2333,6 @@ public void testCreateAndGetTemplates() throws JsonParseException { jsonString, apiToken ); - createTemplateResponse.then().assertThat().statusCode(OK.getStatusCode()) .body("data.name", equalTo("Dataverse template")) .body("data.isDefault", equalTo(true)) @@ -2704,26 +2373,6 @@ public void testCreateAndGetTemplates() throws JsonParseException { getTemplateResponse.then().assertThat().statusCode(UNAUTHORIZED.getStatusCode()); } - @Test - public void testGetStorageDriver() { - Response updatedStorageDriver = UtilIT.getStorageDriver("root", getSuperuserToken(), Boolean.TRUE); - updatedStorageDriver.prettyPrint(); - updatedStorageDriver.then().assertThat() - .body("data.name", CoreMatchers.notNullValue()) - .body("data.type", CoreMatchers.notNullValue()) - .body("data.label", CoreMatchers.notNullValue()) - .body("data.directUpload", CoreMatchers.nullValue()) - .body("data.directDownload", CoreMatchers.nullValue()) - .statusCode(200); - - // Root without default is undefined - updatedStorageDriver = UtilIT.getStorageDriver("root", getSuperuserToken(), null); - updatedStorageDriver.prettyPrint(); - updatedStorageDriver.then().assertThat() - .body("data.name", CoreMatchers.equalTo(DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER)) - .statusCode(200); - } - private String getSuperuserToken() { Response createResponse = UtilIT.createRandomUser(); String adminApiToken = UtilIT.getApiTokenFromResponse(createResponse); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java b/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java index 24625c87ce2..137c2ef4c7b 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java @@ -1,6 +1,5 @@ package edu.harvard.iq.dataverse.api; -import org.hamcrest.CoreMatchers; import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; import software.amazon.awssdk.core.ResponseInputStream; @@ -143,7 +142,7 @@ public void testNonDirectUpload() { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.name", equalTo("undefined")) + .body("data.message", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); @@ -154,9 +153,6 @@ public void testNonDirectUpload() { Response updatedStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); updatedStorageDriver.prettyPrint(); updatedStorageDriver.then().assertThat() - .body("data.type", CoreMatchers.notNullValue()) - .body("data.label", CoreMatchers.notNullValue()) - .body("data.directUpload", CoreMatchers.nullValue()) .statusCode(200); Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); @@ -277,7 +273,7 @@ public void testDirectUpload() { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.name", equalTo("undefined")) + .body("data.message", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); @@ -495,7 +491,7 @@ public void testDirectUploadDetectStataFile() { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.name", equalTo("undefined")) + .body("data.message", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); @@ -693,7 +689,7 @@ public void testDirectUploadWithFileCountLimit() throws JsonParseException { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.name", equalTo("undefined")) + .body("data.message", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java index c11f66aa749..eba8181e566 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java @@ -2850,13 +2850,9 @@ static Response listStorageDrivers(String apiToken) { } static Response getStorageDriver(String dvAlias, String apiToken) { - return getStorageDriver(dvAlias, apiToken, null); - } - static Response getStorageDriver(String dvAlias, String apiToken, Boolean getEffective) { - String params = getEffective != null ? "?getEffective=" + getEffective : ""; return given() .header(API_TOKEN_HTTP_HEADER, apiToken) - .get("/api/admin/dataverse/" + dvAlias + "/storageDriver" + params); + .get("/api/admin/dataverse/" + dvAlias + "/storageDriver"); } static Response setStorageDriver(String dvAlias, String label, String apiToken) { @@ -4464,12 +4460,6 @@ static Response setDatasetStorageDriver(Integer datasetId, String driverLabel, S .put("/api/datasets/" + datasetId + "/storageDriver"); } - static Response getDatasetStorageDriver(Integer datasetId, String apiToken) { - return given() - .header(API_TOKEN_HTTP_HEADER, apiToken) - .get("/api/datasets/" + datasetId + "/storageDriver"); - } - /** GET on /api/admin/savedsearches/list */ static Response getSavedSearchList() { return given().get("/api/admin/savedsearches/list"); @@ -4543,18 +4533,6 @@ public static Response updateDataverseInputLevels(String dataverseAlias, String[ .contentType(ContentType.JSON) .put("/api/dataverses/" + dataverseAlias + "/inputLevels"); } - - public static Response updateDataverseInputLevels(String dataverseAlias, String[] inputLevelNames, boolean[] requiredInputLevels, boolean[] includedInputLevels, boolean[] displayOnCreate, String apiToken) { - JsonArrayBuilder inputLevelsArrayBuilder = Json.createArrayBuilder(); - for (int i = 0; i < inputLevelNames.length; i++) { - inputLevelsArrayBuilder.add(createInputLevelObject(inputLevelNames[i], requiredInputLevels[i], includedInputLevels[i], displayOnCreate[i])); - } - return given() - .header(API_TOKEN_HTTP_HEADER, apiToken) - .body(inputLevelsArrayBuilder.build().toString()) - .contentType(ContentType.JSON) - .put("/api/dataverses/" + dataverseAlias + "/inputLevels"); - } private static JsonObjectBuilder createInputLevelObject(String name, boolean required, boolean include) { return Json.createObjectBuilder() @@ -4562,14 +4540,6 @@ private static JsonObjectBuilder createInputLevelObject(String name, boolean req .add("required", required) .add("include", include); } - - private static JsonObjectBuilder createInputLevelObject(String name, boolean required, boolean include, boolean displayOnCreate) { - return Json.createObjectBuilder() - .add("datasetFieldTypeName", name) - .add("required", required) - .add("include", include) - .add("displayOnCreate", displayOnCreate); - } public static Response getOpenAPI(String accept, String format) { Response response = given() @@ -4758,59 +4728,6 @@ static Response updateDataverseFeaturedItem(long featuredItemId, String apiToken) { return updateDataverseFeaturedItem(featuredItemId, content, displayOrder, keepFile, pathToFile, null, null, apiToken); } - - static Response getLinkableDataverses(String type, String dvObjectId, String apiToken, String searchTerm) { - return getLinkableDataverses(type, dvObjectId, apiToken, searchTerm, false); - } - - static Response getLinkableDataverses(String type, String dvObjectId, String apiToken, String searchTerm, boolean alreadyLinked) { - - String idInPath = dvObjectId; // Assume it's a number to start. - String optionalQueryParam = ""; // If idOrPersistentId is a number we'll just put it in the path. - if (type.equals("dataset")) { - if (!NumberUtils.isCreatable(idInPath)) { - idInPath = ":persistentId"; - if (searchTerm == null || searchTerm.isEmpty() ) { - optionalQueryParam = "?persistentId=" + dvObjectId; - } else { - optionalQueryParam = "&persistentId=" + dvObjectId; - } - } - } - - if (alreadyLinked){ - if (optionalQueryParam.isEmpty() && (searchTerm == null || searchTerm.isEmpty())){ - optionalQueryParam = "?alreadyLinking=true"; - } else { - optionalQueryParam = optionalQueryParam + "&alreadyLinking=true"; - } - } - - if (searchTerm == null) { - if (!apiToken.isEmpty()) { - return given() - .header(API_TOKEN_HTTP_HEADER, apiToken) - .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses" + optionalQueryParam); - - } else { - return given() - .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses" + optionalQueryParam); - } - - } else { - if (!apiToken.isEmpty()) { - return given() - .header(API_TOKEN_HTTP_HEADER, apiToken) - .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses?searchTerm=" + searchTerm + optionalQueryParam); - - } else { - return given() - .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses?searchTerm=" + searchTerm + optionalQueryParam); - } - - } - } - static Response updateDataverseFeaturedItem(long featuredItemId, String content, int displayOrder, diff --git a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java index 2d2fd943d4d..73880b78e7b 100644 --- a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java @@ -307,7 +307,7 @@ public void testCustomOptions() throws CommandException { i++; } - // assertTrue( dftilsDeleted ); we no longer delete when adding new input levels to preserve previously created + assertTrue( dftilsDeleted ); for ( DataverseFieldTypeInputLevel dftil : createdDftils ) { assertEquals( result, dftil.getDataverse() ); } diff --git a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java deleted file mode 100644 index 7f167370c63..00000000000 --- a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java +++ /dev/null @@ -1,163 +0,0 @@ -package edu.harvard.iq.dataverse.engine.command.impl; - -import edu.harvard.iq.dataverse.*; -import edu.harvard.iq.dataverse.authorization.Permission; -import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; -import edu.harvard.iq.dataverse.engine.command.CommandContext; -import edu.harvard.iq.dataverse.engine.command.DataverseRequest; -import edu.harvard.iq.dataverse.engine.command.exception.CommandException; -import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; -import edu.harvard.iq.dataverse.util.BundleUtil; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.extension.ExtendWith; -import org.mockito.Mock; -import org.mockito.Mockito; -import org.mockito.junit.jupiter.MockitoExtension; - -import java.util.Arrays; -import java.util.Collections; -import java.util.List; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertThrows; -import static org.junit.jupiter.api.Assertions.assertTrue; - -@ExtendWith(MockitoExtension.class) -public class GetLinkingDataverseListCommandTest { - - @Mock - private DataverseRequest dataverseRequest; - @Mock - private CommandContext commandContext; - @Mock - private PermissionServiceBean permissionService; - @Mock - private DataverseServiceBean dataverseService; - @Mock - private AuthenticatedUser authenticatedUser; - - @BeforeEach - public void setUp() { - Mockito.when(dataverseRequest.getUser()).thenReturn(authenticatedUser); - } - - @Test - public void execute_shouldThrowException_whenUserIsNotAuthenticated() { - // Arrange - Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(false); - DvObject dvObject = new Dataset(); - GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, dvObject, ""); - - // Act & Assert - assertThrows(IllegalCommandException.class, () -> { - sut.execute(commandContext); - }, BundleUtil.getStringFromBundle("dataverse.link.user")); - } - - @Test - public void execute_shouldReturnPermittedList_forLinkingDataset() throws CommandException { - // Arrange - String searchTerm = "test"; - Dataset datasetToLink = new Dataset(); - List permittedList = Arrays.asList(new Dataverse(), new Dataverse()); - - Mockito.when(commandContext.permissions()).thenReturn(permissionService); - Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); - Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); - Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, searchTerm)) - .thenReturn(permittedList); - Mockito.when(dataverseService.removeUnlinkableDataverses(permittedList, datasetToLink)) - .thenReturn(permittedList); // Assume none are removed for this test - - GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, datasetToLink, searchTerm); - - // Act - List result = sut.execute(commandContext); - - // Assert - assertEquals(2, result.size()); - assertEquals(permittedList, result); - Mockito.verify(permissionService).findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, searchTerm); - Mockito.verify(dataverseService).removeUnlinkableDataverses(permittedList, datasetToLink); - } - - @Test - public void execute_shouldReturnPermittedList_forLinkingDataverse() throws CommandException { - // Arrange - String searchTerm = "test"; - Dataverse dataverseToLink = new Dataverse(); - List permittedList = Collections.singletonList(new Dataverse()); - - Mockito.when(commandContext.permissions()).thenReturn(permissionService); - Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); - Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); - Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataverse, searchTerm)) - .thenReturn(permittedList); - Mockito.when(dataverseService.removeUnlinkableDataverses(permittedList, dataverseToLink)) - .thenReturn(permittedList); - - GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, dataverseToLink, searchTerm); - - // Act - List result = sut.execute(commandContext); - - // Assert - assertEquals(1, result.size()); - assertEquals(permittedList, result); - Mockito.verify(permissionService).findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataverse, searchTerm); - Mockito.verify(dataverseService).removeUnlinkableDataverses(permittedList, dataverseToLink); - } - - @Test - public void execute_shouldUseEmptyString_whenSearchTermIsNull() throws CommandException { - // Arrange - String searchTerm = null; - Dataset datasetToLink = new Dataset(); - List expectedList = Collections.emptyList(); - - Mockito.when(commandContext.permissions()).thenReturn(permissionService); - // if there are none found then the remove unlinkable doesn't get run - // Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); - Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); - // Note: verify that an empty string "" is passed instead of null - Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, "")) - .thenReturn(expectedList); - // if there are none found then the remove unlinkable doesn't get run - // Mockito.when(dataverseService.removeUnlinkableDataverses(expectedList, datasetToLink)).thenReturn(expectedList); - - GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, datasetToLink, searchTerm); - - // Act - List result = sut.execute(commandContext); - - // Assert - assertTrue(result.isEmpty()); - Mockito.verify(permissionService).findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, ""); - } - - @Test - public void execute_shouldReturnEmptyList_whenNoPermittedCollectionsFound() throws CommandException { - // Arrange - String searchTerm = "nonexistent"; - Dataset datasetToLink = new Dataset(); - List emptyList = Collections.emptyList(); - - Mockito.when(commandContext.permissions()).thenReturn(permissionService); - // if there are none found then the remove unlinkable doesn't get run - // Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); - Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); - Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, searchTerm)) - .thenReturn(emptyList); - // if there are none found then the remove unlinkable doesn't get run - // Mockito.when(dataverseService.removeUnlinkableDataverses(emptyList, datasetToLink)).thenReturn(emptyList); - - GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, datasetToLink, searchTerm); - - // Act - List result = sut.execute(commandContext); - - // Assert - assertTrue(result.isEmpty()); - } -} From bf0f63e0519f3be298086880f8ad07cbcd6aafb6 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 15 Aug 2025 13:06:09 -0400 Subject: [PATCH 052/578] add "review" dataset type #11747 --- .../11747-review-dataset-type.md | 3 + doc/sphinx-guides/source/api/native-api.rst | 30 +- .../source/user/dataset-management.rst | 4 +- .../harvard/iq/dataverse/DataCitation.java | 3 + .../iq/dataverse/dataset/DatasetType.java | 1 + .../pidproviders/doi/XmlMetadataTemplate.java | 2 + src/main/java/propertyFiles/Bundle.properties | 1 + .../iq/dataverse/api/DatasetTypesIT.java | 290 ++++++++++++++++++ 8 files changed, 327 insertions(+), 7 deletions(-) create mode 100644 doc/release-notes/11747-review-dataset-type.md diff --git a/doc/release-notes/11747-review-dataset-type.md b/doc/release-notes/11747-review-dataset-type.md new file mode 100644 index 00000000000..aa05d48dc66 --- /dev/null +++ b/doc/release-notes/11747-review-dataset-type.md @@ -0,0 +1,3 @@ +### New Dataset Type: Review + +A new, experimental dataset type called "review" has been added. When this type is published, it will be sent to DataCite as "Other" for resourceTypeGeneral. See #11747. diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index ada3ae5a8af..747df73a712 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -970,8 +970,8 @@ You should expect an HTTP 200 ("OK") response and JSON indicating the database I .. _api-create-dataset-with-type: -Create a Dataset with a Dataset Type (Software, etc.) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Create a Dataset with a Dataset Type (Software, Workflow, Review, etc.) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default, datasets are given the type "dataset" but if your installation had added additional types (see :ref:`api-add-dataset-type`), you can specify the type. @@ -1025,8 +1025,8 @@ Before calling the API, make sure the data files referenced by the ``POST``\ ed .. _import-dataset-with-type: -Import a Dataset with a Dataset Type (Software, etc.) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Import a Dataset with a Dataset Type (Software, Workflow, Review, etc.) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default, datasets are given the type "dataset" but if your installation had added additional types (see :ref:`api-add-dataset-type`), you can specify the type. @@ -3925,7 +3925,27 @@ The fully expanded example above (without environment variables) looks like this Add Dataset Type ^^^^^^^^^^^^^^^^ -Note: Before you add any types of your own, there should be a single type called "dataset". If you add "software" or "workflow", these types will be sent to DataCite (if you use DataCite). Otherwise, the only functionality you gain currently from adding types is an entry in the "Dataset Type" facet but be advised that if you add a type other than "software" or "workflow", you will need to add your new type to your Bundle.properties file for it to appear in Title Case rather than lower case in the "Dataset Type" facet. +Note: Before you add any types of your own, there should be a single type called "dataset". + +Adding certain dataset types will result in a value other than "Dataset" being sent to DataCite (if you use DataCite) as shown in the table below. + +.. list-table:: Values sent to DataCite for resourceTypeGeneral by Dataset Type + :header-rows: 1 + :stub-columns: 1 + :align: left + + * - Dataset Type + - Value sent to DataCite + * - dataset + - Dataset + * - software + - Software + * - workflow + - Workflow + * - review + - Other + +Other than sending a different resourceTypeGeneral to DataCite, the only functionality you gain currently from adding types is an entry in the "Dataset Type" facet but be advised that if you add a type other than "software", "workflow", or "review", you will need to add your new type to your Bundle.properties file for it to appear in Title Case rather than lower case in the "Dataset Type" facet. With all that said, we'll add a "software" type in the example below. This API endpoint is superuser only. The "name" of a type cannot be only digits. Note that this endpoint also allows you to add metadata blocks and available licenses for your new dataset type by adding "linkedMetadataBlocks" and/or "availableLicenses" arrays to your JSON. diff --git a/doc/sphinx-guides/source/user/dataset-management.rst b/doc/sphinx-guides/source/user/dataset-management.rst index d73459969ce..8b443520cc7 100755 --- a/doc/sphinx-guides/source/user/dataset-management.rst +++ b/doc/sphinx-guides/source/user/dataset-management.rst @@ -824,11 +824,11 @@ Dataset Types .. note:: Development of the dataset types feature is ongoing. Please see https://github.com/IQSS/dataverse-pm/issues/307 for details. -Out of the box, all datasets have a dataset type of "dataset". Superusers can add additional types such as "software" or "workflow" using the :ref:`api-add-dataset-type` API endpoint. +Out of the box, all datasets have a dataset type of "dataset". Superusers can add additional types such as "software", "workflow", or "review" using the :ref:`api-add-dataset-type` API endpoint. Once more than one type appears in search results, a facet called "Dataset Type" will appear allowing you to filter down to a certain type. -If your installation is configured to use DataCite as a persistent ID (PID) provider, the appropriate type ("Dataset", "Software", "Workflow") will be sent to DataCite when the dataset is published for those three types. +If your installation is configured to use DataCite as a persistent ID (PID) provider, the appropriate type ("Dataset", "Software", "Workflow", "Review") will be sent to DataCite when the dataset is published for those types. Currently, specifying a type for a dataset can only be done via API and only when the dataset is created. The type can't currently be changed afterward. For details, see the following sections of the API guide: diff --git a/src/main/java/edu/harvard/iq/dataverse/DataCitation.java b/src/main/java/edu/harvard/iq/dataverse/DataCitation.java index 30d9928f59a..1f6380f020f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataCitation.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataCitation.java @@ -740,8 +740,11 @@ public Map getDataCiteMetadata() { public JsonObject getCSLJsonFormat() { CSLItemDataBuilder itemBuilder = new CSLItemDataBuilder(); + // TODO consider making this a switch if (type.equals(DatasetType.DATASET_TYPE_SOFTWARE)) { itemBuilder.type(CSLType.SOFTWARE); + } else if (type.equals(DatasetType.DATASET_TYPE_REVIEW)) { + itemBuilder.type(CSLType.REVIEW); } else { itemBuilder.type(CSLType.DATASET); } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java index c55324f66e3..e437af472e2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java @@ -39,6 +39,7 @@ public class DatasetType implements Serializable { public static final String DATASET_TYPE_DATASET = "dataset"; public static final String DATASET_TYPE_SOFTWARE = "software"; public static final String DATASET_TYPE_WORKFLOW = "workflow"; + public static final String DATASET_TYPE_REVIEW = "review"; public static final String DEFAULT_DATASET_TYPE = DATASET_TYPE_DATASET; @Id diff --git a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java index 9ebb346baf8..efc81d4179f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java +++ b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java @@ -842,6 +842,8 @@ private void writeResourceType(XMLStreamWriter xmlw, DvObject dvObject) throws X case DatasetType.DATASET_TYPE_DATASET -> "Dataset"; case DatasetType.DATASET_TYPE_SOFTWARE -> "Software"; case DatasetType.DATASET_TYPE_WORKFLOW -> "Workflow"; + // "Other" for now but we might ask DataCite to support https://schema.org/CriticReview + case DatasetType.DATASET_TYPE_REVIEW -> "Other"; default -> "Dataset"; }; } diff --git a/src/main/java/propertyFiles/Bundle.properties b/src/main/java/propertyFiles/Bundle.properties index 1016bb96788..0bbf1758dba 100644 --- a/src/main/java/propertyFiles/Bundle.properties +++ b/src/main/java/propertyFiles/Bundle.properties @@ -13,6 +13,7 @@ passwd=Password dataset=Dataset software=Software workflow=Workflow +review=Review # END dataset types datasets=Datasets newDataset=New Dataset diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java index 205725822ff..924950f5e8c 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java @@ -1,6 +1,8 @@ package edu.harvard.iq.dataverse.api; +import static edu.harvard.iq.dataverse.api.ApiConstants.DS_VERSION_LATEST_PUBLISHED; import edu.harvard.iq.dataverse.dataset.DatasetType; +import edu.harvard.iq.dataverse.util.StringUtil; import io.restassured.RestAssured; import io.restassured.path.json.JsonPath; import io.restassured.response.Response; @@ -37,6 +39,7 @@ public static void setUpClass() { UtilIT.setSuperuserStatus(username, true).then().assertThat().statusCode(OK.getStatusCode()); ensureDatasetTypeIsPresent(DatasetType.DATASET_TYPE_SOFTWARE, apiToken); + ensureDatasetTypeIsPresent(DatasetType.DATASET_TYPE_REVIEW, apiToken); ensureDatasetTypeIsPresent(INSTRUMENT, apiToken); } @@ -713,4 +716,291 @@ public void testCreateDatasetWithCustomType() { } + /** + * In this test, there are two users: one who publishes a dataset and + * another who publishes a review of that dataset. + */ + @Test + public void testCreateReview() { + Response createDatasetDepositor = UtilIT.createRandomUser(); + createDatasetDepositor.then().assertThat().statusCode(OK.getStatusCode()); + String apiTokenDepositor = UtilIT.getApiTokenFromResponse(createDatasetDepositor); + + Response createCollectionOfData = UtilIT.createRandomDataverse(apiTokenDepositor); + createCollectionOfData.then().assertThat().statusCode(CREATED.getStatusCode()); + String collectionOfDataAlias = UtilIT.getAliasFromResponse(createCollectionOfData); + + Response createDataset = UtilIT.createRandomDatasetViaNativeApi(collectionOfDataAlias, apiTokenDepositor); + createDataset.then().assertThat().statusCode(CREATED.getStatusCode()); + Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + String datasetPid = JsonPath.from(createDataset.getBody().asString()).getString("data.persistentId"); + + UtilIT.publishDataverseViaNativeApi(collectionOfDataAlias, apiTokenDepositor).then().assertThat().statusCode(OK.getStatusCode()); + UtilIT.publishDatasetViaNativeApi(datasetPid, "major", apiTokenDepositor); + + Response createReviewer = UtilIT.createRandomUser(); + createReviewer.then().assertThat().statusCode(OK.getStatusCode()); + String apiTokenReviewer = UtilIT.getApiTokenFromResponse(createReviewer); + + // We assume the reviewer wants their own collection for reviews. + Response createCollectionOfReviews = UtilIT.createRandomDataverse(apiTokenReviewer); + createCollectionOfReviews.then().assertThat().statusCode(CREATED.getStatusCode()); + String collectionOfReviewsAlias = UtilIT.getAliasFromResponse(createCollectionOfReviews); + + Response datasetMetadataResponse = UtilIT.nativeGet(datasetId, apiTokenReviewer); + datasetMetadataResponse.then().assertThat().statusCode(OK.getStatusCode()); + datasetMetadataResponse.prettyPrint(); + JsonPath datasetMetadata = JsonPath.from(datasetMetadataResponse.body().asString()); + String datasetTitle = datasetMetadata.getString("data.latestVersion.metadataBlocks.citation.fields[0].value"); + String datasetPidUrl = datasetMetadata.getString("data.persistentUrl"); + String datasetPidProtocol = datasetMetadata.getString("data.protocol"); + String datasetPidAuthority = datasetMetadata.getString("data.authority"); + String datasetPidSeparator = datasetMetadata.getString("data.separator"); + String datasetPidIdentifier = datasetMetadata.getString("data.identifier"); + String datasetPidWithoutProtocol = datasetPidAuthority + datasetPidSeparator + datasetPidIdentifier; + + Response getCitation = UtilIT.getDatasetVersionCitation(datasetId, DS_VERSION_LATEST_PUBLISHED, false, apiTokenReviewer); + getCitation.prettyPrint(); + getCitation.then().assertThat().statusCode(OK.getStatusCode()); + String datasetCitationHtml = JsonPath.from(getCitation.getBody().asString()).getString("data.message"); + String datasetCitationText = StringUtil.html2text(datasetCitationHtml); + + JsonObjectBuilder jsonForCreatingReview = Json.createObjectBuilder() + /** + * See above where this type is added to the installation and + * therefore available for use. + */ + .add("datasetType", DatasetType.DATASET_TYPE_REVIEW) + .add("datasetVersion", Json.createObjectBuilder() + .add("license", Json.createObjectBuilder() + .add("name", "CC0 1.0") + .add("uri", "http://creativecommons.org/publicdomain/zero/1.0") + ) + .add("metadataBlocks", Json.createObjectBuilder() + .add("citation", Json.createObjectBuilder() + .add("fields", Json.createArrayBuilder() + .add(Json.createObjectBuilder() + .add("typeName", "title") + .add("value", "Review of " + datasetTitle) + .add("typeClass", "primitive") + .add("multiple", false) + ) + .add(Json.createObjectBuilder() + .add("value", Json.createArrayBuilder() + .add(Json.createObjectBuilder() + .add("authorName", + Json.createObjectBuilder() + .add("value", "Simpson, Homer") + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "authorName")) + ) + ) + .add("typeClass", "compound") + .add("multiple", true) + .add("typeName", "author") + ) + .add(Json.createObjectBuilder() + .add("value", Json.createArrayBuilder() + .add(Json.createObjectBuilder() + .add("datasetContactEmail", + Json.createObjectBuilder() + .add("value", "hsimpson@mailinator.com") + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "datasetContactEmail")) + ) + ) + .add("typeClass", "compound") + .add("multiple", true) + .add("typeName", "datasetContact") + ) + .add(Json.createObjectBuilder() + .add("value", Json.createArrayBuilder() + .add(Json.createObjectBuilder() + .add("dsDescriptionValue", + Json.createObjectBuilder() + .add("value", "This is a review of a dataset.") + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "dsDescriptionValue")) + ) + ) + .add("typeClass", "compound") + .add("multiple", true) + .add("typeName", "dsDescription") + ) + .add(Json.createObjectBuilder() + .add("value", Json.createArrayBuilder() + .add("Other") + ) + .add("typeClass", "controlledVocabulary") + .add("multiple", true) + .add("typeName", "subject") + ) + /** + * Related Dataset. + * + * The tooltip for Related + * Dataset says "Information, + * such as a persistent ID or + * citation, about a related + * dataset, such as previous + * research on the Dataset's + * subject". + * + * For now we'll add multiple + * forms so we can discuss and + * decide which one to use. + * + * Also, we are well aware that + * there is a custom metadata + * block called + * "relatedDatasetsV2" at + * https://github.com/vera/related-datasets-cvoc + * that we hope to play with + * soon. + */ + .add(Json.createObjectBuilder() + .add("value", Json.createArrayBuilder() + .add(datasetPid) + .add(datasetPidUrl) + .add(datasetCitationText) + .add(datasetCitationHtml) + ) + .add("typeClass", "primitive") + .add("multiple", true) + .add("typeName", "relatedDatasets") + ) + /** + * Related Publication. + * + * Related Dataset is more + * appropriate than Related + * Publication but we're adding + * Related Publication (twice, + * with plain text and HTML + * citations) because unlike + * Related Dataset, Related + * Publication lets us send + * publicationRelationType, + * which gets sent to DataCite + * as "References" or "Cites" or + * whatever. + * + * (Ideally, we'd send "Reviews" + * but we get this error: + * + * "Error parsing Json: Invalid + * controlled vocabulary in + * compound field Value + * 'Reviews' does not exist in + * type + * 'publicationRelationType';") + */ + .add(Json.createObjectBuilder() + .add("value", Json.createArrayBuilder() + // Plain text citation version + .add(Json.createObjectBuilder() + .add("publicationRelationType", + Json.createObjectBuilder() + .add("value", "References") + // .add("value", "Reviews") + .add("typeClass", "controlledVocabulary") + .add("multiple", false) + .add("typeName", "publicationRelationType") + ) + .add("publicationCitation", + Json.createObjectBuilder() + .add("value", datasetCitationHtml) + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "publicationCitation") + ) + .add("publicationIDType", + Json.createObjectBuilder() + .add("value", datasetPidProtocol) + .add("typeClass", "controlledVocabulary") + .add("multiple", false) + .add("typeName", "publicationIDType") + ) + .add("publicationIDNumber", + Json.createObjectBuilder() + .add("value", datasetPidWithoutProtocol) + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "publicationIDNumber") + ) + .add("publicationURL", + Json.createObjectBuilder() + .add("value", datasetPidUrl) + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "publicationURL") + ) + ) + // HTML citation version + .add(Json.createObjectBuilder() + .add("publicationRelationType", + Json.createObjectBuilder() + .add("value", "References") + // .add("value", "Reviews") + .add("typeClass", "controlledVocabulary") + .add("multiple", false) + .add("typeName", "publicationRelationType") + ) + .add("publicationCitation", + Json.createObjectBuilder() + .add("value", datasetCitationText) + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "publicationCitation") + ) + .add("publicationIDType", + Json.createObjectBuilder() + .add("value", datasetPidProtocol) + .add("typeClass", "controlledVocabulary") + .add("multiple", false) + .add("typeName", "publicationIDType") + ) + .add("publicationIDNumber", + Json.createObjectBuilder() + .add("value", datasetPidWithoutProtocol) + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "publicationIDNumber") + ) + .add("publicationURL", + Json.createObjectBuilder() + .add("value", datasetPidUrl) + .add("typeClass", "primitive") + .add("multiple", false) + .add("typeName", "publicationURL") + ) + ) + ) + .add("typeClass", "compound") + .add("multiple", true) + .add("typeName", "publication") + ) + ) + ) + )); + + Response createReview = UtilIT.createDataset(collectionOfReviewsAlias, jsonForCreatingReview, apiTokenReviewer); + createReview.prettyPrint(); + createReview.then().assertThat().statusCode(CREATED.getStatusCode()); + Integer reviewId = UtilIT.getDatasetIdFromResponse(createReview); + String reviewPid = JsonPath.from(createReview.getBody().asString()).getString("data.persistentId"); + + Response getReviewMetadata = UtilIT.nativeGet(reviewId, apiTokenReviewer); + getReviewMetadata.prettyPrint(); + getReviewMetadata.then().assertThat().statusCode(OK.getStatusCode()); + String datasetType = JsonPath.from(getReviewMetadata.getBody().asString()).getString("data.datasetType"); + assertEquals("review", datasetType); + + UtilIT.publishDataverseViaNativeApi(collectionOfReviewsAlias, apiTokenReviewer).then().assertThat().statusCode(OK.getStatusCode()); + UtilIT.publishDatasetViaNativeApi(reviewPid, "major", apiTokenReviewer).then().assertThat().statusCode(OK.getStatusCode()); + } + } From 0a48cc8017bdf7b6b4967cfe42cd67ffbb26b219 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 26 Aug 2025 10:12:42 -0400 Subject: [PATCH 053/578] only populate Related Dataset and only HTML version #11747 Also, rework Javadoc comment. --- .../iq/dataverse/api/DatasetTypesIT.java | 154 ++---------------- 1 file changed, 17 insertions(+), 137 deletions(-) diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java index 924950f5e8c..42926aad758 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java @@ -765,6 +765,23 @@ public void testCreateReview() { String datasetCitationHtml = JsonPath.from(getCitation.getBody().asString()).getString("data.message"); String datasetCitationText = StringUtil.html2text(datasetCitationHtml); + /** + * We are added the HTML version of a Related Dataset. We like the HTML + * version because both JSF and the SPA render the DOI link as a + * clickable link. + * + * The tooltip for Related Dataset says "Information, such as a + * persistent ID or citation, about a related dataset, such as previous + * research on the Dataset's subject". + * + * We are aware that there is a custom metadata block called + * "relatedDatasetsV2" at https://github.com/vera/related-datasets-cvoc + * that we have been playing with. We especially like that relationships + * can be expressed between the current object (a review) and the + * related dataset. This is simlar to how "Related Publication" works. + * See also discussion at + * https://dataverse.zulipchat.com/#narrow/channel/379673-dev/topic/Improved.20.22Related.20datasets.22/near/534969036 + */ JsonObjectBuilder jsonForCreatingReview = Json.createObjectBuilder() /** * See above where this type is added to the installation and @@ -838,151 +855,14 @@ public void testCreateReview() { .add("multiple", true) .add("typeName", "subject") ) - /** - * Related Dataset. - * - * The tooltip for Related - * Dataset says "Information, - * such as a persistent ID or - * citation, about a related - * dataset, such as previous - * research on the Dataset's - * subject". - * - * For now we'll add multiple - * forms so we can discuss and - * decide which one to use. - * - * Also, we are well aware that - * there is a custom metadata - * block called - * "relatedDatasetsV2" at - * https://github.com/vera/related-datasets-cvoc - * that we hope to play with - * soon. - */ .add(Json.createObjectBuilder() .add("value", Json.createArrayBuilder() - .add(datasetPid) - .add(datasetPidUrl) - .add(datasetCitationText) .add(datasetCitationHtml) ) .add("typeClass", "primitive") .add("multiple", true) .add("typeName", "relatedDatasets") ) - /** - * Related Publication. - * - * Related Dataset is more - * appropriate than Related - * Publication but we're adding - * Related Publication (twice, - * with plain text and HTML - * citations) because unlike - * Related Dataset, Related - * Publication lets us send - * publicationRelationType, - * which gets sent to DataCite - * as "References" or "Cites" or - * whatever. - * - * (Ideally, we'd send "Reviews" - * but we get this error: - * - * "Error parsing Json: Invalid - * controlled vocabulary in - * compound field Value - * 'Reviews' does not exist in - * type - * 'publicationRelationType';") - */ - .add(Json.createObjectBuilder() - .add("value", Json.createArrayBuilder() - // Plain text citation version - .add(Json.createObjectBuilder() - .add("publicationRelationType", - Json.createObjectBuilder() - .add("value", "References") - // .add("value", "Reviews") - .add("typeClass", "controlledVocabulary") - .add("multiple", false) - .add("typeName", "publicationRelationType") - ) - .add("publicationCitation", - Json.createObjectBuilder() - .add("value", datasetCitationHtml) - .add("typeClass", "primitive") - .add("multiple", false) - .add("typeName", "publicationCitation") - ) - .add("publicationIDType", - Json.createObjectBuilder() - .add("value", datasetPidProtocol) - .add("typeClass", "controlledVocabulary") - .add("multiple", false) - .add("typeName", "publicationIDType") - ) - .add("publicationIDNumber", - Json.createObjectBuilder() - .add("value", datasetPidWithoutProtocol) - .add("typeClass", "primitive") - .add("multiple", false) - .add("typeName", "publicationIDNumber") - ) - .add("publicationURL", - Json.createObjectBuilder() - .add("value", datasetPidUrl) - .add("typeClass", "primitive") - .add("multiple", false) - .add("typeName", "publicationURL") - ) - ) - // HTML citation version - .add(Json.createObjectBuilder() - .add("publicationRelationType", - Json.createObjectBuilder() - .add("value", "References") - // .add("value", "Reviews") - .add("typeClass", "controlledVocabulary") - .add("multiple", false) - .add("typeName", "publicationRelationType") - ) - .add("publicationCitation", - Json.createObjectBuilder() - .add("value", datasetCitationText) - .add("typeClass", "primitive") - .add("multiple", false) - .add("typeName", "publicationCitation") - ) - .add("publicationIDType", - Json.createObjectBuilder() - .add("value", datasetPidProtocol) - .add("typeClass", "controlledVocabulary") - .add("multiple", false) - .add("typeName", "publicationIDType") - ) - .add("publicationIDNumber", - Json.createObjectBuilder() - .add("value", datasetPidWithoutProtocol) - .add("typeClass", "primitive") - .add("multiple", false) - .add("typeName", "publicationIDNumber") - ) - .add("publicationURL", - Json.createObjectBuilder() - .add("value", datasetPidUrl) - .add("typeClass", "primitive") - .add("multiple", false) - .add("typeName", "publicationURL") - ) - ) - ) - .add("typeClass", "compound") - .add("multiple", true) - .add("typeName", "publication") - ) ) ) )); From d2879078257926632db025d6e9e10642b751617a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 26 Aug 2025 10:24:28 -0400 Subject: [PATCH 054/578] add blocks for Trusted Data and Repo Characteristics #11747 --- doc/sphinx-guides/source/user/appendix.rst | 2 + .../repositorycharacteristics.tsv | 8 +++ .../trusteddatadimensionsintensities.tsv | 61 +++++++++++++++++++ 3 files changed, 71 insertions(+) create mode 100644 scripts/api/data/metadatablocks/repositorycharacteristics.tsv create mode 100644 scripts/api/data/metadatablocks/trusteddatadimensionsintensities.tsv diff --git a/doc/sphinx-guides/source/user/appendix.rst b/doc/sphinx-guides/source/user/appendix.rst index d1c46a93fdf..99a01fa41b0 100755 --- a/doc/sphinx-guides/source/user/appendix.rst +++ b/doc/sphinx-guides/source/user/appendix.rst @@ -43,6 +43,8 @@ Unlike supported metadata, experimental metadata is not enabled by default in a - Computational Workflow Metadata (`see .tsv `__): adapted from `Bioschemas Computational Workflow Profile, version 1.0 `__ and `Codemeta `__. - Archival Metadata (`see .tsv `__): Enables repositories to register metadata relating to the potential archiving of the dataset at a depositor archive, whether that be your own institutional archive or an external archive, i.e. a historical archive. - Local Contexts Metadata (`see .tsv `__): Supports integration with the `Local Contexts `__ platform, enabling the use of Traditional Knowledge and Biocultural Labels, and Notices. For more information on setup and configuration, see :doc:`../installation/localcontexts`. +- Trusted Data Dimensions and Intensities (`see .tsv `__): Enables repositories to indicate dimensions of trust. +- Repository Characteristics (`see .tsv `__): Details related to the security, sustainability, and certifications of the repository. Please note: these custom metadata schemas are not included in the Solr schema for indexing by default, you will need to add them as necessary for your custom metadata blocks. See "Update the Solr Schema" in :doc:`../admin/metadatacustomization`. diff --git a/scripts/api/data/metadatablocks/repositorycharacteristics.tsv b/scripts/api/data/metadatablocks/repositorycharacteristics.tsv new file mode 100644 index 00000000000..3951ebb146f --- /dev/null +++ b/scripts/api/data/metadatablocks/repositorycharacteristics.tsv @@ -0,0 +1,8 @@ +#metadataBlock name dataverseAlias displayName blockURI + repositorycharacteristics Repository Characteristics +#datasetField name title description watermark fieldType displayOrder displayFormat advancedSearchField allowControlledVocabulary allowmultiples facetable displayoncreate required parent metadatablock_id termURI + repositoryOrHosting Repository or Hosting Where and how the data are stored (e.g. institutional repository, cloud) and the oversight and management policies that ensure ethical and transparent handling of the data textbox 0 FALSE FALSE FALSE FALSE FALSE FALSE repositorycharacteristics + securityAndTechnicalSafeguards Security and Technical Safeguards The mechanisms that the repository or hosting platform uses to prevent tampering with or unauthorized access to the data textbox 1 FALSE FALSE FALSE FALSE FALSE FALSE repositorycharacteristics + sustainabilityAndLongevity Sustainability and Longevity The strategies that the repository or hosting platform uses to support the long-term preservation and maintenance of the data textbox 2 FALSE FALSE FALSE FALSE FALSE FALSE repositorycharacteristics + standardsCertification Standards Certification The certification of the repository or hosting platform (e.g. CoreTrustSeal) textbox 3 FALSE FALSE FALSE FALSE FALSE FALSE repositorycharacteristics +#controlledVocabulary DatasetField Value identifier displayOrder \ No newline at end of file diff --git a/scripts/api/data/metadatablocks/trusteddatadimensionsintensities.tsv b/scripts/api/data/metadatablocks/trusteddatadimensionsintensities.tsv new file mode 100644 index 00000000000..44af02ee2be --- /dev/null +++ b/scripts/api/data/metadatablocks/trusteddatadimensionsintensities.tsv @@ -0,0 +1,61 @@ +#metadataBlock name dataverseAlias displayName blockURI + trusteddatadimensionsintensities Trusted Data Dimensions and Intensities +#datasetField name title description watermark fieldType displayOrder displayFormat advancedSearchField allowControlledVocabulary allowmultiples facetable displayoncreate required parent metadatablock_id termURI + reviewTarget Review Target The type of research object reviewed text 0 TRUE TRUE FALSE TRUE FALSE FALSE trusteddatadimensionsintensities + authorAndProvenance Author and Provenance The level of trust in the data creators and in other provenance information text 1 TRUE TRUE FALSE TRUE FALSE FALSE trusteddatadimensionsintensities + integrityAndUsability Integrity and Usability The level of trust in the accuracy, completeness, and ease of use of the data text 2 TRUE TRUE FALSE TRUE FALSE FALSE trusteddatadimensionsintensities + fitnessForScopeAndContextualRelevance Fitness for Scope and Contextual Relevance The level of trust in the suitability of the data for specific contexts, questions, or policy applications text 3 TRUE TRUE FALSE TRUE FALSE FALSE trusteddatadimensionsintensities + licensingAndLegalClarity Licensing and Legal Clarity The level of trust in the explicitness of the data’s usage rights and their compliance with relevant laws and regulations text 4 TRUE TRUE FALSE TRUE FALSE FALSE trusteddatadimensionsintensities + transparencyOfMethodsAndDocumentation Transparency of Methods and Documentation The level of trust in the clarity of the descriptions of data collection and processing methods text 5 TRUE TRUE FALSE TRUE FALSE FALSE trusteddatadimensionsintensities + biasEquityAndRepresentativeness Bias, Equity, and Representativeness The level of trust in the inclusivity and fairness of the coverage of the data text 6 TRUE TRUE FALSE TRUE FALSE FALSE trusteddatadimensionsintensities +#controlledVocabulary DatasetField Value identifier displayOrder + reviewTarget Audiovisual 0 + reviewTarget Award 1 + reviewTarget Book 2 + reviewTarget Book Chapter 3 + reviewTarget Collection 4 + reviewTarget Computational Notebook 5 + reviewTarget Conference Paper 6 + reviewTarget Conference Proceeding 7 + reviewTarget DataPaper 8 + reviewTarget Dataset 9 + reviewTarget Dissertation 10 + reviewTarget Event 11 + reviewTarget Image 12 + reviewTarget Interactive Resource 13 + reviewTarget Instrument 14 + reviewTarget Journal 15 + reviewTarget Journal Article 16 + reviewTarget Model 17 + reviewTarget Output Management Plan 18 + reviewTarget Peer Review 19 + reviewTarget Physical Object 20 + reviewTarget Preprint 21 + reviewTarget Project 22 + reviewTarget Report 23 + reviewTarget Service 24 + reviewTarget Software 25 + reviewTarget Sound 26 + reviewTarget Standard 27 + reviewTarget Study Registration 28 + reviewTarget Text 29 + reviewTarget Workflow 30 + reviewTarget Other 31 + authorAndProvenance Low 0 + authorAndProvenance Medium 1 + authorAndProvenance High 2 + integrityAndUsability Low 0 + integrityAndUsability Medium 1 + integrityAndUsability High 2 + fitnessForScopeAndContextualRelevance Low 0 + fitnessForScopeAndContextualRelevance Medium 1 + fitnessForScopeAndContextualRelevance High 2 + licensingAndLegalClarity Low 0 + licensingAndLegalClarity Medium 1 + licensingAndLegalClarity High 2 + transparencyOfMethodsAndDocumentation Low 0 + transparencyOfMethodsAndDocumentation Medium 1 + transparencyOfMethodsAndDocumentation High 2 + biasEquityAndRepresentativeness Low 0 + biasEquityAndRepresentativeness Medium 1 + biasEquityAndRepresentativeness High 2 \ No newline at end of file From c938a7a51d1de91f4b3acd2feb95865d763b59aa Mon Sep 17 00:00:00 2001 From: Julian Gautier Date: Tue, 2 Sep 2025 09:21:50 -0400 Subject: [PATCH 055/578] Adding properties files for two metadata blocks Adding properties files for the repositorycharacteristics.tsv and trusteddatadimensionsintensities.tsv metadata blocks --- .../repositorycharacteristics.properties | 15 ++++ ...rusteddatadimensionsintensities.properties | 74 +++++++++++++++++++ 2 files changed, 89 insertions(+) create mode 100644 src/main/java/propertyFiles/repositorycharacteristics.properties create mode 100644 src/main/java/propertyFiles/trusteddatadimensionsintensities.properties diff --git a/src/main/java/propertyFiles/repositorycharacteristics.properties b/src/main/java/propertyFiles/repositorycharacteristics.properties new file mode 100644 index 00000000000..6d07ae3cdac --- /dev/null +++ b/src/main/java/propertyFiles/repositorycharacteristics.properties @@ -0,0 +1,15 @@ +metadatablock.name=repositorycharacteristics +metadatablock.displayName=Repository Characteristics +metadatablock.displayFacet= +datasetfieldtype.repositoryOrHosting.title=Repository or Hosting +datasetfieldtype.securityAndTechnicalSafeguards.title=Security and Technical Safeguards +datasetfieldtype.sustainabilityAndLongevity.title=Sustainability and Longevity +datasetfieldtype.standardsCertification.title=Standards Certification +datasetfieldtype.repositoryOrHosting.description=Where and how the data are stored (e.g. institutional repository, cloud) and the oversight and management policies that ensure ethical and transparent handling of the data +datasetfieldtype.securityAndTechnicalSafeguards.description=The mechanisms that the repository or hosting platform uses to prevent tampering with or unauthorized access to the data +datasetfieldtype.sustainabilityAndLongevity.description=The strategies that the repository or hosting platform uses to support the long-term preservation and maintenance of the data +datasetfieldtype.standardsCertification.description=The certification of the repository or hosting platform (e.g. CoreTrustSeal) +datasetfieldtype.repositoryOrHosting.watermark= +datasetfieldtype.securityAndTechnicalSafeguards.watermark= +datasetfieldtype.sustainabilityAndLongevity.watermark= +datasetfieldtype.standardsCertification.watermark= \ No newline at end of file diff --git a/src/main/java/propertyFiles/trusteddatadimensionsintensities.properties b/src/main/java/propertyFiles/trusteddatadimensionsintensities.properties new file mode 100644 index 00000000000..a06cea18234 --- /dev/null +++ b/src/main/java/propertyFiles/trusteddatadimensionsintensities.properties @@ -0,0 +1,74 @@ +metadatablock.name=trusteddatadimensionsintensities +metadatablock.displayName=Trusted Data Dimensions and Intensities +metadatablock.displayFacet= +datasetfieldtype.reviewTarget.title=Review Target +datasetfieldtype.authorAndProvenance.title=Author and Provenance +datasetfieldtype.integrityAndUsability.title=Integrity and Usability +datasetfieldtype.fitnessForScopeAndContextualRelevance.title=Fitness for Scope and Contextual Relevance +datasetfieldtype.licensingAndLegalClarity.title=Licensing and Legal Clarity +datasetfieldtype.transparencyOfMethodsAndDocumentation.title=Transparency of Methods and Documentation +datasetfieldtype.biasEquityAndRepresentativeness.title=Bias, Equity, and Representativeness +datasetfieldtype.reviewTarget.description=The type of research object reviewed +datasetfieldtype.authorAndProvenance.description=The level of trust in the data creators and in other provenance information +datasetfieldtype.integrityAndUsability.description=The level of trust in the accuracy, completeness, and ease of use of the data +datasetfieldtype.fitnessForScopeAndContextualRelevance.description=The level of trust in the suitability of the data for specific contexts, questions, or policy applications +datasetfieldtype.licensingAndLegalClarity.description=The level of trust in the explicitness of the data’s usage rights and their compliance with relevant laws and regulations +datasetfieldtype.transparencyOfMethodsAndDocumentation.description=The level of trust in the clarity of the descriptions of data collection and processing methods +datasetfieldtype.biasEquityAndRepresentativeness.description=The level of trust in the inclusivity and fairness of the coverage of the data +datasetfieldtype.reviewTarget.watermark= +datasetfieldtype.authorAndProvenance.watermark= +datasetfieldtype.integrityAndUsability.watermark= +datasetfieldtype.fitnessForScopeAndContextualRelevance.watermark= +datasetfieldtype.licensingAndLegalClarity.watermark= +datasetfieldtype.transparencyOfMethodsAndDocumentation.watermark= +datasetfieldtype.biasEquityAndRepresentativeness.watermark= +controlledvocabulary.reviewTarget.audiovisual=Audiovisual +controlledvocabulary.reviewTarget.award=Award +controlledvocabulary.reviewTarget.book=Book +controlledvocabulary.reviewTarget.book_chapter=Book Chapter +controlledvocabulary.reviewTarget.collection=Collection +controlledvocabulary.reviewTarget.computational_notebook=Computational Notebook +controlledvocabulary.reviewTarget.conference_paper=Conference Paper +controlledvocabulary.reviewTarget.conference_proceeding=Conference Proceeding +controlledvocabulary.reviewTarget.datapaper=DataPaper +controlledvocabulary.reviewTarget.dataset=Dataset +controlledvocabulary.reviewTarget.dissertation=Dissertation +controlledvocabulary.reviewTarget.event=Event +controlledvocabulary.reviewTarget.image=Image +controlledvocabulary.reviewTarget.interactive_resource=Interactive Resource +controlledvocabulary.reviewTarget.instrument=Instrument +controlledvocabulary.reviewTarget.journal=Journal +controlledvocabulary.reviewTarget.journal_article=Journal Article +controlledvocabulary.reviewTarget.model=Model +controlledvocabulary.reviewTarget.output_management_plan=Output Management Plan +controlledvocabulary.reviewTarget.peer_review=Peer Review +controlledvocabulary.reviewTarget.physical_object=Physical Object +controlledvocabulary.reviewTarget.preprint=Preprint +controlledvocabulary.reviewTarget.project=Project +controlledvocabulary.reviewTarget.report=Report +controlledvocabulary.reviewTarget.service=Service +controlledvocabulary.reviewTarget.software=Software +controlledvocabulary.reviewTarget.sound=Sound +controlledvocabulary.reviewTarget.standard=Standard +controlledvocabulary.reviewTarget.study_registration=Study Registration +controlledvocabulary.reviewTarget.text=Text +controlledvocabulary.reviewTarget.workflow=Workflow +controlledvocabulary.reviewTarget.other=Other +controlledvocabulary.authorAndProvenance.low=Low +controlledvocabulary.authorAndProvenance.medium=Medium +controlledvocabulary.authorAndProvenance.high=High +controlledvocabulary.integrityAndUsability.low=Low +controlledvocabulary.integrityAndUsability.medium=Medium +controlledvocabulary.integrityAndUsability.high=High +controlledvocabulary.fitnessForScopeAndContextualRelevance.low=Low +controlledvocabulary.fitnessForScopeAndContextualRelevance.medium=Medium +controlledvocabulary.fitnessForScopeAndContextualRelevance.high=High +controlledvocabulary.licensingAndLegalClarity.low=Low +controlledvocabulary.licensingAndLegalClarity.medium=Medium +controlledvocabulary.licensingAndLegalClarity.high=High +controlledvocabulary.transparencyOfMethodsAndDocumentation.low=Low +controlledvocabulary.transparencyOfMethodsAndDocumentation.medium=Medium +controlledvocabulary.transparencyOfMethodsAndDocumentation.high=High +controlledvocabulary.biasEquityAndRepresentativeness.low=Low +controlledvocabulary.biasEquityAndRepresentativeness.medium=Medium +controlledvocabulary.biasEquityAndRepresentativeness.high=High \ No newline at end of file From 1c7379390ffd5fd203dffd19fcc834bc879278d3 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 8 Oct 2025 12:55:40 -0400 Subject: [PATCH 056/578] add displayName for datasetType #11747 #11887 --- .../iq/dataverse/dataset/DatasetType.java | 18 ++++++++++++++++++ src/main/resources/db/migration/V6.8.0.1.sql | 6 ++++++ 2 files changed, 24 insertions(+) create mode 100644 src/main/resources/db/migration/V6.8.0.1.sql diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java index e437af472e2..36066afd577 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java @@ -46,10 +46,19 @@ public class DatasetType implements Serializable { @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; + /** + * Machine readable name to use via API. + */ // Any constraints? @Pattern regexp? @Column(nullable = false) private String name; + /** + * Human readable name to show in the UI. + */ + @Column(nullable = false) + private String displayName; + /** * The metadata blocks this dataset type is linked to. */ @@ -81,6 +90,14 @@ public void setName(String name) { this.name = name; } + public String getDisplayName() { + return displayName; + } + + public void setDisplayName(String displayName) { + this.displayName = displayName; + } + public List getMetadataBlocks() { return metadataBlocks; } @@ -109,6 +126,7 @@ public JsonObjectBuilder toJson() { return Json.createObjectBuilder() .add("id", getId()) .add("name", getName()) + .add("displayName", getDisplayName()) .add("linkedMetadataBlocks", linkedMetadataBlocks) .add("availableLicenses", availableLicenses); } diff --git a/src/main/resources/db/migration/V6.8.0.1.sql b/src/main/resources/db/migration/V6.8.0.1.sql new file mode 100644 index 00000000000..352ce14f1fb --- /dev/null +++ b/src/main/resources/db/migration/V6.8.0.1.sql @@ -0,0 +1,6 @@ +-- Add displayname column to datasettype table +ALTER TABLE datasettype ADD COLUMN IF NOT EXISTS displayname VARCHAR(255); +-- Set displayname for dataset +UPDATE datasettype SET displayname = 'Dataset' WHERE name = 'dataset'; +-- Make displayname required +ALTER TABLE datasettype ALTER COLUMN displayname SET NOT NULL; \ No newline at end of file From 9138524e40e62e9eb5b60e7866cac3d059e8cff0 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 8 Oct 2025 15:02:29 -0400 Subject: [PATCH 057/578] have displayname default to empty string #11747 This is so an earlier script (V6.3.0.3.sql) continues to work. It inserts a row without a displayName. --- .../java/edu/harvard/iq/dataverse/dataset/DatasetType.java | 2 +- src/main/resources/db/migration/V6.8.0.1.sql | 6 ++---- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java index 36066afd577..528132665fc 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java @@ -56,7 +56,7 @@ public class DatasetType implements Serializable { /** * Human readable name to show in the UI. */ - @Column(nullable = false) + @Column(nullable = false, columnDefinition = "VARCHAR(255) DEFAULT ''") private String displayName; /** diff --git a/src/main/resources/db/migration/V6.8.0.1.sql b/src/main/resources/db/migration/V6.8.0.1.sql index 352ce14f1fb..158278843ab 100644 --- a/src/main/resources/db/migration/V6.8.0.1.sql +++ b/src/main/resources/db/migration/V6.8.0.1.sql @@ -1,6 +1,4 @@ -- Add displayname column to datasettype table -ALTER TABLE datasettype ADD COLUMN IF NOT EXISTS displayname VARCHAR(255); +ALTER TABLE datasettype ADD COLUMN IF NOT EXISTS displayname VARCHAR(255) NOT NULL DEFAULT ''; -- Set displayname for dataset -UPDATE datasettype SET displayname = 'Dataset' WHERE name = 'dataset'; --- Make displayname required -ALTER TABLE datasettype ALTER COLUMN displayname SET NOT NULL; \ No newline at end of file +UPDATE datasettype SET displayname = 'Dataset' WHERE name = 'dataset'; \ No newline at end of file From e6447b77ecc908fed1f3dd1e7a852f4bdce84b72 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 8 Oct 2025 16:29:30 -0400 Subject: [PATCH 058/578] when creating dataset types, allow displayName #11747 --- .../harvard/iq/dataverse/api/Datasets.java | 6 +++++ .../iq/dataverse/api/DatasetTypesIT.java | 25 ++++++++++++++++--- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index 3fd490c38bc..1cbce4e1806 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -5732,6 +5732,7 @@ public Response addDatasetType(@Context ContainerRequestContext crc, String json } String nameIn = null; + String displayNameIn = null; JsonArrayBuilder datasetTypesAfter = Json.createArrayBuilder(); List metadataBlocksToSave = new ArrayList<>(); @@ -5740,6 +5741,7 @@ public Response addDatasetType(@Context ContainerRequestContext crc, String json try { JsonObject datasetTypeObj = JsonUtil.getJsonObject(jsonIn); nameIn = datasetTypeObj.getString("name"); + displayNameIn = datasetTypeObj.getString("displayName", null); JsonArray arr = datasetTypeObj.getJsonArray("linkedMetadataBlocks"); if (arr != null && !arr.isEmpty()) { @@ -5777,6 +5779,9 @@ public Response addDatasetType(@Context ContainerRequestContext crc, String json if (nameIn == null) { return error(BAD_REQUEST, "A name for the dataset type is required"); } + if (displayNameIn == null) { + return error(BAD_REQUEST, "A displayName for the dataset type is required"); + } if (StringUtils.isNumeric(nameIn)) { // getDatasetTypes supports id or name so we don't want a names that looks like an id return error(BAD_REQUEST, "The name of the type cannot be only digits."); @@ -5785,6 +5790,7 @@ public Response addDatasetType(@Context ContainerRequestContext crc, String json try { DatasetType datasetType = new DatasetType(); datasetType.setName(nameIn); + datasetType.setDisplayName(displayNameIn); datasetType.setMetadataBlocks(metadataBlocksToSave); datasetType.setLicenses(licensesToSave); DatasetType saved = datasetTypeSvc.save(datasetType); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java index 42926aad758..1eee1c7811f 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java @@ -58,12 +58,20 @@ private static void ensureDatasetTypeIsPresent(String datasetType, String apiTok return; } System.out.println("The " + datasetType + "type wasn't found. Create it."); - String jsonIn = Json.createObjectBuilder().add("name", datasetType).build().toString(); + String displayName = capitalize(datasetType); + String jsonIn = Json.createObjectBuilder() + .add("name", datasetType) + .add("displayName", displayName) + .build().toString(); Response typeAdded = UtilIT.addDatasetType(jsonIn, apiToken); typeAdded.prettyPrint(); typeAdded.then().assertThat().statusCode(OK.getStatusCode()); } + private static String capitalize(String stringIn) { + return stringIn.substring(0, 1).toUpperCase() + stringIn.substring(1); + } + @Test public void testCreateSoftwareDatasetNative() { Response createUser = UtilIT.createRandomUser(); @@ -262,7 +270,11 @@ public void testAddAndDeleteDatasetType() { //Avoid all-numeric names (which are not allowed) String randomName = "A" + UUID.randomUUID().toString().substring(0, 8); - String jsonIn = Json.createObjectBuilder().add("name", randomName).build().toString(); + String displayName = capitalize(randomName); + String jsonIn = Json.createObjectBuilder() + .add("name", randomName) + .add("displayName", displayName) + .build().toString(); System.out.println("adding type with name " + randomName); Response typeAdded = UtilIT.addDatasetType(jsonIn, apiToken); @@ -295,6 +307,7 @@ public void testAddDatasetTypeWithMDBLicense(){ JsonObjectBuilder job = Json.createObjectBuilder(); job.add("name", "testDatasetType"); + job.add("displayName", "testDatasetType"); job.add("linkedMetadataBlocks", Json.createArrayBuilder().add("geospatial")); job.add("availableLicenses", Json.createArrayBuilder().add("CC0 1.0")); @@ -352,6 +365,7 @@ public void testUpdateDatasetTypeWithLicense(){ JsonObjectBuilder job = Json.createObjectBuilder(); job.add("name", "testDatasetType"); + job.add("displayName", "testDatasetType"); Response typeAdded = UtilIT.addDatasetType(job.build(), apiToken); typeAdded.prettyPrint(); @@ -420,7 +434,11 @@ public void testUpdateDatasetTypeLinksWithMetadataBlocks() { //Avoid all-numeric names (which are not allowed) String randomName = "zzz" + UUID.randomUUID().toString().substring(0, 8); - String jsonIn = Json.createObjectBuilder().add("name", randomName).build().toString(); + String displayName = capitalize(randomName); + String jsonIn = Json.createObjectBuilder() + .add("name", randomName) + .add("displayName", displayName) + .build().toString(); System.out.println("adding type with name " + randomName); Response typeAdded = UtilIT.addDatasetType(jsonIn, apiToken); @@ -655,6 +673,7 @@ public void testCreateDatasetWithCustomType() { JsonObjectBuilder job = Json.createObjectBuilder(); job.add("name", "testDatasetType"); + job.add("displayName", "testDatasetType"); job.add("linkedMetadataBlocks", Json.createArrayBuilder().add("geospatial")); job.add("availableLicenses", Json.createArrayBuilder().add("CC0 1.0")); From 99fcede6aa3a31d96e3a3933b8b2c0ffff227d88 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Thu, 23 Oct 2025 16:49:51 -0400 Subject: [PATCH 059/578] allow i18n of displayName for dataset types #11747 --- .../harvard/iq/dataverse/api/Datasets.java | 19 ++++-- .../iq/dataverse/dataset/DatasetType.java | 39 ++++++++++- .../harvard/iq/dataverse/i18n/i18nUtil.java | 25 +++++++ .../propertyFiles/datasetTypes.properties | 8 +++ .../iq/dataverse/api/DatasetTypesIT.java | 65 +++++++++++++++++++ .../edu/harvard/iq/dataverse/api/UtilIT.java | 25 +++++-- .../iq/dataverse/i18n/i18nUtilTest.java | 44 +++++++++++++ 7 files changed, 211 insertions(+), 14 deletions(-) create mode 100644 src/main/java/edu/harvard/iq/dataverse/i18n/i18nUtil.java create mode 100644 src/main/java/propertyFiles/datasetTypes.properties create mode 100644 src/test/java/edu/harvard/iq/dataverse/i18n/i18nUtilTest.java diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index 1cbce4e1806..5494738d257 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -34,6 +34,7 @@ import edu.harvard.iq.dataverse.externaltools.ExternalToolHandler; import edu.harvard.iq.dataverse.globus.GlobusServiceBean; import edu.harvard.iq.dataverse.globus.GlobusUtil; +import edu.harvard.iq.dataverse.i18n.i18nUtil; import edu.harvard.iq.dataverse.ingest.IngestServiceBean; import edu.harvard.iq.dataverse.ingest.IngestUtil; import edu.harvard.iq.dataverse.makedatacount.*; @@ -99,13 +100,12 @@ import java.util.stream.Collectors; import static edu.harvard.iq.dataverse.api.ApiConstants.*; -import edu.harvard.iq.dataverse.dataset.DatasetType; -import edu.harvard.iq.dataverse.dataset.DatasetTypeServiceBean; import edu.harvard.iq.dataverse.license.License; import static edu.harvard.iq.dataverse.util.json.JsonPrinter.*; import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; +import static jakarta.ws.rs.core.HttpHeaders.ACCEPT_LANGUAGE; import static jakarta.ws.rs.core.Response.Status.BAD_REQUEST; import static jakarta.ws.rs.core.Response.Status.NOT_FOUND; import static jakarta.ws.rs.core.Response.Status.FORBIDDEN; @@ -5684,17 +5684,19 @@ public Response resetPidGenerator(@Context ContainerRequestContext crc, @PathPar @GET @Path("datasetTypes") - public Response getDatasetTypes() { + public Response getDatasetTypes(@HeaderParam(ACCEPT_LANGUAGE) String acceptLanguage) { + Locale locale = i18nUtil.parseAcceptLanguageHeader(acceptLanguage); JsonArrayBuilder jab = Json.createArrayBuilder(); for (DatasetType datasetType : datasetTypeSvc.listAll()) { - jab.add(datasetType.toJson()); + jab.add(datasetType.toJson(locale)); } return ok(jab); } @GET @Path("datasetTypes/{idOrName}") - public Response getDatasetTypes(@PathParam("idOrName") String idOrName) { + public Response getDatasetTypes(@PathParam("idOrName") String idOrName, @HeaderParam(ACCEPT_LANGUAGE) String acceptLanguage) { + Locale locale = i18nUtil.parseAcceptLanguageHeader(acceptLanguage); DatasetType datasetType = null; if (StringUtils.isNumeric(idOrName)) { try { @@ -5707,7 +5709,7 @@ public Response getDatasetTypes(@PathParam("idOrName") String idOrName) { datasetType = datasetTypeSvc.getByName(idOrName); } if (datasetType != null) { - return ok(datasetType.toJson()); + return ok(datasetType.toJson(locale)); } else { return error(NOT_FOUND, "Could not find a dataset type with name " + idOrName); } @@ -5796,7 +5798,10 @@ public Response addDatasetType(@Context ContainerRequestContext crc, String json DatasetType saved = datasetTypeSvc.save(datasetType); Long typeId = saved.getId(); String name = saved.getName(); - return ok(saved.toJson()); + // Locale is null because when creating the dataset type we are relying entirely + // on the database. The new dataset type has not yet been localized in a + // properties file. + return ok(saved.toJson(null)); } catch (WrappedResponse ex) { return error(BAD_REQUEST, ex.getMessage()); } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java index 528132665fc..14d97c0824b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetType.java @@ -2,6 +2,7 @@ import edu.harvard.iq.dataverse.MetadataBlock; import edu.harvard.iq.dataverse.license.License; +import edu.harvard.iq.dataverse.util.BundleUtil; import jakarta.json.Json; import jakarta.json.JsonArrayBuilder; import jakarta.json.JsonObjectBuilder; @@ -19,6 +20,9 @@ import java.io.Serializable; import java.util.ArrayList; import java.util.List; +import java.util.Locale; +import java.util.MissingResourceException; +import java.util.logging.Logger; @NamedQueries({ @NamedQuery(name = "DatasetType.findAll", @@ -36,6 +40,8 @@ public class DatasetType implements Serializable { + private static final Logger logger = Logger.getLogger(DatasetType.class.getCanonicalName()); + public static final String DATASET_TYPE_DATASET = "dataset"; public static final String DATASET_TYPE_SOFTWARE = "software"; public static final String DATASET_TYPE_WORKFLOW = "workflow"; @@ -90,6 +96,10 @@ public void setName(String name) { this.name = name; } + /** + * In most cases, you should call the getDisplayName(locale) version. This is + * here in case you really want the value from the database. + */ public String getDisplayName() { return displayName; } @@ -114,7 +124,7 @@ public void setLicenses(List licenses) { this.licenses = licenses; } - public JsonObjectBuilder toJson() { + public JsonObjectBuilder toJson(Locale locale) { JsonArrayBuilder linkedMetadataBlocks = Json.createArrayBuilder(); for (MetadataBlock metadataBlock : this.getMetadataBlocks()) { linkedMetadataBlocks.add(metadataBlock.getName()); @@ -126,9 +136,34 @@ public JsonObjectBuilder toJson() { return Json.createObjectBuilder() .add("id", getId()) .add("name", getName()) - .add("displayName", getDisplayName()) + .add("displayName", getDisplayName(locale)) .add("linkedMetadataBlocks", linkedMetadataBlocks) .add("availableLicenses", availableLicenses); } + public String getDisplayName(Locale locale) { + logger.fine("Getting display name for dataset type " + name + " and locale " + locale); + if (locale == null) { + logger.fine("Locale is null, returning default display name: " + displayName); + return displayName; + } + if (locale.getLanguage().isBlank()) { + logger.fine("Locale couldn't be parsed, returning default display name: " + displayName); + return displayName; + } + if (locale.getLanguage().equals(Locale.ENGLISH.getLanguage())) { + // This is here to prevent looking up datasetTypes_en.properties, which doesn't exist. + // The English strings are in datasetTypes.properties (no _en). + logger.fine("Locale is English, returning default display name: " + displayName); + return displayName; + } + String propertiesFile = "datasetTypes_" + locale.toLanguageTag() + ".properties"; + try { + logger.fine("Looking up " + name + ".displayName in " + propertiesFile); + return BundleUtil.getStringFromPropertyFile(name + ".displayName", "datasetTypes", locale); + } catch (MissingResourceException e) { + logger.warning(name + ".displayName missing from " + propertiesFile + " (or file does not exist). Returning English version."); + return displayName; + } + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/i18n/i18nUtil.java b/src/main/java/edu/harvard/iq/dataverse/i18n/i18nUtil.java new file mode 100644 index 00000000000..11f855256e1 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/i18n/i18nUtil.java @@ -0,0 +1,25 @@ +package edu.harvard.iq.dataverse.i18n; + +import java.util.List; +import java.util.Locale; + +public class i18nUtil { + + /** + * @param acceptLanguageHeader The Accept-Language header value such as + * "Accept-Language: en-US,en;q=0.5" + * @return The first Locale or null. + */ + public static Locale parseAcceptLanguageHeader(String acceptLanguageHeader) { + if (acceptLanguageHeader == null || acceptLanguageHeader.isEmpty()) { + return null; + } + List list = Locale.LanguageRange.parse(acceptLanguageHeader); + if (list.isEmpty()) { + return null; + } + Locale.LanguageRange languageRange = list.get(0); + return Locale.forLanguageTag(languageRange.getRange()); + } + +} diff --git a/src/main/java/propertyFiles/datasetTypes.properties b/src/main/java/propertyFiles/datasetTypes.properties new file mode 100644 index 00000000000..8b650b63f11 --- /dev/null +++ b/src/main/java/propertyFiles/datasetTypes.properties @@ -0,0 +1,8 @@ +# This file contains the strings for dataset types that can be +# translated into other languages (French, Spanish, etc.). +# Only the default dataset type (dataset) is included, as an example. +# If you add additional dataset types (e.g. software), you don't +# need to add them to this file if you are only running in English. +# However, if you are running in additional languages, you should +# add the additional dataset types to this file. +dataset.displayName=Dataset \ No newline at end of file diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java index 1eee1c7811f..3e9066b0c50 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetTypesIT.java @@ -3,6 +3,7 @@ import static edu.harvard.iq.dataverse.api.ApiConstants.DS_VERSION_LATEST_PUBLISHED; import edu.harvard.iq.dataverse.dataset.DatasetType; import edu.harvard.iq.dataverse.util.StringUtil; +import static io.restassured.path.json.JsonPath.with; import io.restassured.RestAssured; import io.restassured.path.json.JsonPath; import io.restassured.response.Response; @@ -12,6 +13,8 @@ import static jakarta.ws.rs.core.Response.Status.CREATED; import static jakarta.ws.rs.core.Response.Status.FORBIDDEN; import static jakarta.ws.rs.core.Response.Status.OK; +import java.util.List; +import java.util.Map; import java.util.UUID; import org.hamcrest.CoreMatchers; import static org.hamcrest.CoreMatchers.containsString; @@ -902,4 +905,66 @@ public void testCreateReview() { UtilIT.publishDatasetViaNativeApi(reviewPid, "major", apiTokenReviewer).then().assertThat().statusCode(OK.getStatusCode()); } + @Test + public void testInternationalization() { + Response getDatasetType = UtilIT.getDatasetType("software"); + getDatasetType.prettyPrint(); + getDatasetType.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.name", is("software")) + .body("data.displayName", is("Software")); + + // https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Language + getDatasetType = UtilIT.getDatasetType("software", "en-US,en;q=0.5"); + getDatasetType.prettyPrint(); + getDatasetType.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.name", is("software")) + .body("data.displayName", is("Software")); + + getDatasetType = UtilIT.getDatasetType("software", "en-US"); + getDatasetType.prettyPrint(); + getDatasetType.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.name", is("software")) + .body("data.displayName", is("Software")); + + getDatasetType = UtilIT.getDatasetType("software", ""); + getDatasetType.prettyPrint(); + getDatasetType.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.name", is("software")) + .body("data.displayName", is("Software")); + + boolean i18nIsConfigured = false; + if (!i18nIsConfigured) { + System.out.println("i18n is not configured; skipping test of non-English languages"); + return; + } + + getDatasetType = UtilIT.getDatasetType("software", "fr-CA,fr;q=0.8,en-US;q=0.6,en;q=0.4"); + getDatasetType.prettyPrint(); + getDatasetType.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.name", is("software")) + .body("data.displayName", is("Logiciel")); + + getDatasetType = UtilIT.getDatasetTypes("fr-CA,fr;q=0.8,en-US;q=0.6,en;q=0.4"); + getDatasetType.prettyPrint(); + getDatasetType.then().assertThat() + .statusCode(OK.getStatusCode()); + + // Messy but the only way we've figured out ¯\_(ツ)_/¯ + List> dataset = with(getDatasetType.body().asString()).param("dataset", "dataset") + .getList("data.findAll { data -> data.name == dataset }"); + Map firstDataset = dataset.get(0); + assertEquals("Ensemble de données", firstDataset.get("displayName")); + + List> instrument = with(getDatasetType.body().asString()).param("instrument", "instrument") + .getList("data.findAll { data -> data.name == instrument }"); + Map firstInstrument = instrument.get(0); + // Instrument isn't translated in the French properties file; should fall back to English + assertEquals("Instrument", firstInstrument.get("displayName")); + } + } diff --git a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java index e80fabc0137..7ba5205833d 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java @@ -16,6 +16,7 @@ import jakarta.json.JsonObject; import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; +import static jakarta.ws.rs.core.HttpHeaders.ACCEPT_LANGUAGE; import static jakarta.ws.rs.core.Response.Status.CREATED; import java.nio.charset.StandardCharsets; @@ -4622,14 +4623,28 @@ static Response listDataverseInputLevels(String dataverseAlias, String apiToken) } public static Response getDatasetTypes() { - Response response = given() - .get("/api/datasets/datasetTypes"); - return response; + return getDatasetTypes(null); + } + + public static Response getDatasetTypes(String acceptLanguage) { + RequestSpecification requestSpecification = given(); + if (acceptLanguage != null) { + https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Language + requestSpecification.header(ACCEPT_LANGUAGE, acceptLanguage); + } + return requestSpecification.get("/api/datasets/datasetTypes"); } static Response getDatasetType(String idOrName) { - return given() - .get("/api/datasets/datasetTypes/" + idOrName); + return getDatasetType(idOrName, null); + } + + static Response getDatasetType(String idOrName, String acceptLanguage) { + RequestSpecification requestSpecification = given(); + if (acceptLanguage != null) { + requestSpecification.header(ACCEPT_LANGUAGE, acceptLanguage); + } + return requestSpecification.get("/api/datasets/datasetTypes/" + idOrName); } static Response addDatasetType(String jsonIn, String apiToken) { diff --git a/src/test/java/edu/harvard/iq/dataverse/i18n/i18nUtilTest.java b/src/test/java/edu/harvard/iq/dataverse/i18n/i18nUtilTest.java new file mode 100644 index 00000000000..99e20e52433 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/i18n/i18nUtilTest.java @@ -0,0 +1,44 @@ +package edu.harvard.iq.dataverse.i18n; + +import org.junit.jupiter.api.Test; +import static org.junit.jupiter.api.Assertions.*; +import java.util.Locale; + +public class i18nUtilTest { + + @Test + void testParseAcceptLanguageHeader_singleLanguage() { + Locale locale = i18nUtil.parseAcceptLanguageHeader("en-US"); + assertEquals(Locale.forLanguageTag("en-US"), locale); + } + + @Test + void testParseAcceptLanguageHeader_singleLanguageWithQ() { + Locale locale = i18nUtil.parseAcceptLanguageHeader("en-US,en;q=0.5"); + assertEquals(Locale.forLanguageTag("en-US"), locale); + } + + @Test + void testParseAcceptLanguageHeader_multipleLanguages() { + Locale locale = i18nUtil.parseAcceptLanguageHeader("fr-CA,fr;q=0.8,en-US;q=0.6,en;q=0.4"); + assertEquals(Locale.forLanguageTag("fr-CA"), locale); + } + + @Test + void testParseAcceptLanguageHeader_emptyHeader() { + Locale locale = i18nUtil.parseAcceptLanguageHeader(""); + assertNull(locale); + } + + @Test + void testParseAcceptLanguageHeader_nullHeader() { + Locale locale = i18nUtil.parseAcceptLanguageHeader(null); + assertNull(locale); + } + + @Test + void testParseAcceptLanguageHeader_invalidHeader() { + Locale locale = i18nUtil.parseAcceptLanguageHeader("invalid-header"); + assertEquals(Locale.forLanguageTag("invalid-header"), locale); + } +} \ No newline at end of file From 58328ec9b3dc4045b33b0bf550eae0e2fd067458 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 23 Oct 2025 16:55:15 -0400 Subject: [PATCH 060/578] Reapply "Merge branch '11670-notification-of-moved-datasets' of https://github.com/IQSS/dataverse into 11670-notification-of-moved-datasets" This reverts commit 75d0e90588fbf7df57a96f31f8c89c9f4f23463b. --- .github/actions/setup-maven/action.yml | 2 +- .github/workflows/codeql.yml | 4 +- .github/workflows/container_app_pr.yml | 4 +- .github/workflows/container_maintenance.yml | 6 +- .github/workflows/deploy_beta_testing.yml | 2 +- .github/workflows/maven_cache_management.yml | 2 +- .github/workflows/maven_unit_test.yml | 6 +- .github/workflows/spi_release.yml | 10 +- conf/keycloak/builtin-users-spi/pom.xml | 2 +- conf/mdc/counter_weekly.sh | 92 +++++ .../11387-modify-input-level-api.md | 3 + .../11695-change-api-get-storage-driver.md | 12 + .../11710-get-available-dataverses-api.md | 5 + .../11752-croissant-restricted.md | 13 + .../11766-new-io.gdcc.dataverse-spi.md | 2 + .../11777-MDC-citation-api-improvement.md | 7 + .../11783-Curation-Status-fixes.md | 1 + doc/release-notes/11800-qb-ra-fixes.md | 2 + .../11822-faster-permission-indexing.md | 1 + .../source/admin/dataverses-datasets.rst | 4 + .../source/admin/make-data-count.rst | 2 + doc/sphinx-guides/source/admin/monitoring.rst | 2 +- doc/sphinx-guides/source/api/changelog.rst | 6 + .../source/api/external-tools.rst | 4 +- doc/sphinx-guides/source/api/native-api.rst | 47 ++- .../source/container/app-image.rst | 2 +- .../source/container/running/demo.rst | 2 +- .../source/contributor/documentation.md | 84 ++-- .../source/developers/deployment.rst | 4 +- .../developers/making-library-releases.rst | 26 ++ .../source/developers/making-releases.rst | 105 ++++- .../source/developers/testing.rst | 16 +- .../source/installation/config.rst | 18 +- .../source/installation/prep.rst | 2 +- .../source/qa/testing-approach.md | 2 +- makefile | 6 +- modules/container-base/README.md | 2 +- modules/container-configbaker/README.md | 2 +- modules/dataverse-spi/pom.xml | 10 +- .../io/gdcc/spi/export/ExportDataContext.java | 61 +++ .../io/gdcc/spi/export/ExportDataOption.java | 51 +++ .../gdcc/spi/export/ExportDataProvider.java | 45 ++- .../java/io/gdcc/spi/export/Exporter.java | 1 - src/main/docker/README.md | 2 +- .../iq/dataverse/DatasetLinkingDataverse.java | 13 + .../dataverse/DatasetLinkingServiceBean.java | 44 ++- .../dataverse/DataverseLinkingDataverse.java | 11 + .../DataverseLinkingServiceBean.java | 37 +- .../iq/dataverse/DataverseServiceBean.java | 78 +++- .../edu/harvard/iq/dataverse/FilePage.java | 5 + .../iq/dataverse/PermissionServiceBean.java | 96 ++++- .../iq/dataverse/api/AbstractApiBean.java | 10 +- .../edu/harvard/iq/dataverse/api/Admin.java | 12 +- .../harvard/iq/dataverse/api/Datasets.java | 4 +- .../dataverse/api/DataverseFeaturedItems.java | 2 +- .../harvard/iq/dataverse/api/Dataverses.java | 39 +- .../iq/dataverse/api/MakeDataCountApi.java | 172 ++++++-- .../providers/shib/ShibServiceBean.java | 4 +- .../impl/AbstractWriteDataverseCommand.java | 8 +- .../CuratePublishedDatasetVersionCommand.java | 25 +- .../impl/GetLinkingDataverseListCommand.java | 113 ++++++ .../iq/dataverse/export/ExportService.java | 1 + .../DatasetExternalCitationsServiceBean.java | 4 +- .../pidproviders/doi/XmlMetadataTemplate.java | 11 +- .../search/SolrIndexServiceBean.java | 211 +++++----- .../iq/dataverse/settings/JvmSettings.java | 4 + .../iq/dataverse/util/json/JsonPrinter.java | 50 ++- .../webapp/WEB-INF/glassfish-resources.xml | 10 + src/main/webapp/dataset.xhtml | 2 +- src/main/webapp/file.xhtml | 2 + .../guestbook-terms-popup-fragment.xhtml | 2 +- src/main/webapp/resources/css/structure.css | 3 + .../harvard/iq/dataverse/api/DatasetsIT.java | 118 +++++- .../iq/dataverse/api/DataversesIT.java | 373 +++++++++++++++++- .../harvard/iq/dataverse/api/S3AccessIT.java | 12 +- .../edu/harvard/iq/dataverse/api/UtilIT.java | 85 +++- .../impl/CreateDataverseCommandTest.java | 2 +- .../GetLinkingDataverseListCommandTest.java | 163 ++++++++ 78 files changed, 2088 insertions(+), 320 deletions(-) create mode 100644 conf/mdc/counter_weekly.sh create mode 100644 doc/release-notes/11387-modify-input-level-api.md create mode 100644 doc/release-notes/11695-change-api-get-storage-driver.md create mode 100644 doc/release-notes/11710-get-available-dataverses-api.md create mode 100644 doc/release-notes/11752-croissant-restricted.md create mode 100644 doc/release-notes/11766-new-io.gdcc.dataverse-spi.md create mode 100644 doc/release-notes/11777-MDC-citation-api-improvement.md create mode 100644 doc/release-notes/11783-Curation-Status-fixes.md create mode 100644 doc/release-notes/11800-qb-ra-fixes.md create mode 100644 doc/release-notes/11822-faster-permission-indexing.md create mode 100644 modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java create mode 100644 modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java create mode 100644 src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java create mode 100644 src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java diff --git a/.github/actions/setup-maven/action.yml b/.github/actions/setup-maven/action.yml index e3e0de47329..62ec7020b5b 100644 --- a/.github/actions/setup-maven/action.yml +++ b/.github/actions/setup-maven/action.yml @@ -23,7 +23,7 @@ runs: echo "JAVA_VERSION=$(grep '' ${GITHUB_WORKSPACE}/modules/dataverse-parent/pom.xml | cut -f2 -d'>' | cut -f1 -d'<')" | tee -a ${GITHUB_ENV} - name: Set up JDK ${{ env.JAVA_VERSION }} id: setup-java - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: java-version: ${{ env.JAVA_VERSION }} distribution: 'temurin' diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 105469139ec..907452f4614 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -71,7 +71,7 @@ jobs: # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL - uses: github/codeql-action/init@v3 + uses: github/codeql-action/init@v4 with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} @@ -99,6 +99,6 @@ jobs: exit 1 - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v3 + uses: github/codeql-action/analyze@v4 with: category: "/language:${{matrix.language}}" diff --git a/.github/workflows/container_app_pr.yml b/.github/workflows/container_app_pr.yml index a5dddc755d1..898b46c2652 100644 --- a/.github/workflows/container_app_pr.yml +++ b/.github/workflows/container_app_pr.yml @@ -23,7 +23,7 @@ jobs: - uses: actions/checkout@v5 with: ref: 'refs/pull/${{ github.event.client_payload.pull_request.number }}/merge' - - uses: actions/setup-java@v4 + - uses: actions/setup-java@v5 with: java-version: "17" distribution: 'adopt' @@ -86,7 +86,7 @@ jobs: :ship: [See on GHCR](https://github.com/orgs/gdcc/packages/container). Use by referencing with full name as printed above, mind the registry name. # Leave a note when things have gone sideways - - uses: peter-evans/create-or-update-comment@v4 + - uses: peter-evans/create-or-update-comment@v5 if: ${{ failure() }} with: issue-number: ${{ github.event.client_payload.pull_request.number }} diff --git a/.github/workflows/container_maintenance.yml b/.github/workflows/container_maintenance.yml index 142363cbe1a..d863f838881 100644 --- a/.github/workflows/container_maintenance.yml +++ b/.github/workflows/container_maintenance.yml @@ -218,7 +218,7 @@ jobs: cat "./modules/container-base/README.md" - name: Push description to DockerHub for base image if: ${{ ! inputs.dry_run && ! inputs.damp_run && toJSON(needs.base-image.outputs.rebuilt_images) != '[]' }} - uses: peter-evans/dockerhub-description@v4 + uses: peter-evans/dockerhub-description@v5 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} @@ -243,7 +243,7 @@ jobs: cat "./src/main/docker/README.md" - name: Push description to DockerHub for application image if: ${{ ! inputs.dry_run && ! inputs.damp_run && toJSON(needs.application-image.outputs.rebuilt_images) != '[]' }} - uses: peter-evans/dockerhub-description@v4 + uses: peter-evans/dockerhub-description@v5 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} @@ -268,7 +268,7 @@ jobs: cat "./modules/container-configbaker/README.md" - name: Push description to DockerHub for config baker image if: ${{ ! inputs.dry_run && ! inputs.damp_run && toJSON(needs.configbaker-image.outputs.rebuilt_images) != '[]' }} - uses: peter-evans/dockerhub-description@v4 + uses: peter-evans/dockerhub-description@v5 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} diff --git a/.github/workflows/deploy_beta_testing.yml b/.github/workflows/deploy_beta_testing.yml index 0e060113ba0..bf4a90d7fb4 100644 --- a/.github/workflows/deploy_beta_testing.yml +++ b/.github/workflows/deploy_beta_testing.yml @@ -16,7 +16,7 @@ jobs: steps: - uses: actions/checkout@v5 - - uses: actions/setup-java@v4 + - uses: actions/setup-java@v5 with: distribution: 'zulu' java-version: '17' diff --git a/.github/workflows/maven_cache_management.yml b/.github/workflows/maven_cache_management.yml index f266b804534..6bfb567c90b 100644 --- a/.github/workflows/maven_cache_management.yml +++ b/.github/workflows/maven_cache_management.yml @@ -36,7 +36,7 @@ jobs: - name: Determine Java version from Parent POM run: echo "JAVA_VERSION=$(grep '' modules/dataverse-parent/pom.xml | cut -f2 -d'>' | cut -f1 -d'<')" >> ${GITHUB_ENV} - name: Set up JDK ${{ env.JAVA_VERSION }} - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: java-version: ${{ env.JAVA_VERSION }} distribution: temurin diff --git a/.github/workflows/maven_unit_test.yml b/.github/workflows/maven_unit_test.yml index a416d5323f0..5857b4489db 100644 --- a/.github/workflows/maven_unit_test.yml +++ b/.github/workflows/maven_unit_test.yml @@ -39,7 +39,7 @@ jobs: # Basic setup chores - uses: actions/checkout@v5 - name: Set up JDK ${{ matrix.jdk }} - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: java-version: ${{ matrix.jdk }} distribution: temurin @@ -105,7 +105,7 @@ jobs: # Basic setup chores - uses: actions/checkout@v5 - name: Set up JDK ${{ matrix.jdk }} - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: java-version: ${{ matrix.jdk }} distribution: temurin @@ -138,7 +138,7 @@ jobs: # TODO: As part of #10618 change to setup-maven custom action # Basic setup chores - uses: actions/checkout@v5 - - uses: actions/setup-java@v4 + - uses: actions/setup-java@v5 with: java-version: '17' distribution: temurin diff --git a/.github/workflows/spi_release.yml b/.github/workflows/spi_release.yml index 9dc722c5992..378e6ff9b67 100644 --- a/.github/workflows/spi_release.yml +++ b/.github/workflows/spi_release.yml @@ -38,11 +38,11 @@ jobs: if: github.event_name == 'pull_request' && needs.check-secrets.outputs.available == 'true' steps: - uses: actions/checkout@v5 - - uses: actions/setup-java@v4 + - uses: actions/setup-java@v5 with: java-version: '17' distribution: 'adopt' - server-id: ossrh + server-id: central server-username: MAVEN_USERNAME server-password: MAVEN_PASSWORD - uses: actions/cache@v4 @@ -64,7 +64,7 @@ jobs: if: github.event_name == 'push' && needs.check-secrets.outputs.available == 'true' steps: - uses: actions/checkout@v5 - - uses: actions/setup-java@v4 + - uses: actions/setup-java@v5 with: java-version: '17' distribution: 'adopt' @@ -76,11 +76,11 @@ jobs: # Running setup-java again overwrites the settings.xml - IT'S MANDATORY TO DO THIS SECOND SETUP!!! - name: Set up Maven Central Repository - uses: actions/setup-java@v4 + uses: actions/setup-java@v5 with: java-version: '17' distribution: 'adopt' - server-id: ossrh + server-id: central server-username: MAVEN_USERNAME server-password: MAVEN_PASSWORD gpg-private-key: ${{ secrets.DATAVERSEBOT_GPG_KEY }} diff --git a/conf/keycloak/builtin-users-spi/pom.xml b/conf/keycloak/builtin-users-spi/pom.xml index 36cf6548d01..2a730621f85 100644 --- a/conf/keycloak/builtin-users-spi/pom.xml +++ b/conf/keycloak/builtin-users-spi/pom.xml @@ -100,7 +100,7 @@ - 26.3.2 + 26.3.4 17 3.2.0 0.4 diff --git a/conf/mdc/counter_weekly.sh b/conf/mdc/counter_weekly.sh new file mode 100644 index 00000000000..67cb5df2af2 --- /dev/null +++ b/conf/mdc/counter_weekly.sh @@ -0,0 +1,92 @@ +#!/bin/sh +#counter_weekly.sh + +# This script iterates through all published Datasets in all Dataverses and calls the Make Data Count API to update their citations from DataCite +# Note: Requires curl and jq for parsing JSON responses form curl + +# A recursive method to process each Dataverse +processDV () { +echo "Processing Dataverse ID#: $1" + +#Call the Dataverse API to get the contents of the Dataverse (without credentials, this will only list published datasets and dataverses +DVCONTENTS=$(curl -s http://localhost:8080/api/dataverses/$1/contents) + +# Iterate over all datasets, pulling the value of their DOIs (as part of the persistentUrl) from the json returned +for subds in $(echo "${DVCONTENTS}" | jq -r '.data[] | select(.type == "dataset") | .persistentUrl'); do + +#The authority/identifier are preceded by a protocol/host, i.e. https://doi.org/ +DOI=`expr "$subds" : '.*:\/\/\doi\.org\/\(.*\)'` + +# Call the Dataverse API for this dataset and capture both the response and HTTP status code +HTTP_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "http://localhost:8080/api/admin/makeDataCount/:persistentId/updateCitationsForDataset?persistentId=doi:$DOI") + +# Extract the HTTP status code from the last line +HTTP_STATUS=$(echo "$HTTP_RESPONSE" | tail -n1) +# Extract the response body (everything except the last line) +RESPONSE_BODY=$(echo "$HTTP_RESPONSE" | sed '$d') + +# Check the HTTP status code and report accordingly +case $HTTP_STATUS in + 200) + # Successfully queued + # Extract status from the nested data object + STATUS=$(echo "$RESPONSE_BODY" | jq -r '.data.status') + + # Extract message from the nested data object + if echo "$RESPONSE_BODY" | jq -e '.data.message' > /dev/null 2>&1 && [ "$(echo "$RESPONSE_BODY" | jq -r '.data.message')" != "null" ]; then + MESSAGE=$(echo "$RESPONSE_BODY" | jq -r '.data.message') + echo "[SUCCESS] doi:$DOI - $STATUS: $MESSAGE" + else + # If message is missing or null, just show the status + echo "[SUCCESS] doi:$DOI - $STATUS: Citation update queued" + fi + ;; + 400) + # Bad request + if echo "$RESPONSE_BODY" | jq -e '.message' > /dev/null 2>&1; then + ERROR=$(echo "$RESPONSE_BODY" | jq -r '.message') + echo "[ERROR 400] doi:$DOI - Bad request: $ERROR" + else + echo "[ERROR 400] doi:$DOI - Bad request" + fi + ;; + 404) + # Not found + if echo "$RESPONSE_BODY" | jq -e '.message' > /dev/null 2>&1; then + ERROR=$(echo "$RESPONSE_BODY" | jq -r '.message') + echo "[ERROR 404] doi:$DOI - Not found: $ERROR" + else + echo "[ERROR 404] doi:$DOI - Not found" + fi + ;; + 503) + # Service unavailable (queue full) + if echo "$RESPONSE_BODY" | jq -e '.message' > /dev/null 2>&1; then + ERROR=$(echo "$RESPONSE_BODY" | jq -r '.message') + echo "[ERROR 503] doi:$DOI - Service unavailable: $ERROR" + elif echo "$RESPONSE_BODY" | jq -e '.data.message' > /dev/null 2>&1; then + ERROR=$(echo "$RESPONSE_BODY" | jq -r '.data.message') + echo "[ERROR 503] doi:$DOI - Service unavailable: $ERROR" + else + echo "[ERROR 503] doi:$DOI - Service unavailable: Queue is full" + fi + ;; + *) + # Other error + echo "[ERROR $HTTP_STATUS] doi:$DOI - Unexpected error" + echo "Response: $RESPONSE_BODY" + ;; +esac + +done + +# Now iterate over any child Dataverses and recursively process them +for subdv in $(echo "${DVCONTENTS}" | jq -r '.data[] | select(.type == "dataverse") | .id'); do +echo $subdv +processDV $subdv +done + +} + +# Call the function on the root dataverse to start processing +processDV 1 \ No newline at end of file diff --git a/doc/release-notes/11387-modify-input-level-api.md b/doc/release-notes/11387-modify-input-level-api.md new file mode 100644 index 00000000000..ca18eabb515 --- /dev/null +++ b/doc/release-notes/11387-modify-input-level-api.md @@ -0,0 +1,3 @@ +### Update Collection Input Level API Changed + +- This endpoint will no longer delete the custom input levels previously modified for the given collection. In order to update a previously modified custom input level, it must be included in the JSON provided to the api. diff --git a/doc/release-notes/11695-change-api-get-storage-driver.md b/doc/release-notes/11695-change-api-get-storage-driver.md new file mode 100644 index 00000000000..aa993232f4d --- /dev/null +++ b/doc/release-notes/11695-change-api-get-storage-driver.md @@ -0,0 +1,12 @@ +## Get Dataset/Dataverse Storage Driver API + +### Changed Json response - breaking change! + +The API for getting the Storage Driver info has been changed/extended. +/api/datasets/{identifier}/storageDriver +/api/admin/dataverse/{dataverse-alias}/storageDriver +changed "message" to "name" and added "type" and "label" + +Also added query param for /api/admin/dataverse/{dataverse-alias}/storageDriver?getEffective=true to recurse the chain of parents to find the effective storageDriver + +See also [the guides](https://dataverse-guide--11664.org.readthedocs.build/en/11664/api/native-api.html#configure-a-dataset-to-store-all-new-files-in-a-specific-file-store), #11695, and #11664. diff --git a/doc/release-notes/11710-get-available-dataverses-api.md b/doc/release-notes/11710-get-available-dataverses-api.md new file mode 100644 index 00000000000..ac33c581848 --- /dev/null +++ b/doc/release-notes/11710-get-available-dataverses-api.md @@ -0,0 +1,5 @@ +### New API endpoint for retrieving a list of Dataverse Collections to which a given Dataset or Dataverse Collection may be linked + +-The end point also takes in a search term which currently must be part of the collections' names. +-The user calling this API must have Link Dataset or Link Dataverse permission on the Dataverse Collections returned. +-If the Collection has already been linked to the given Dataset or Collection, it will not be returned. diff --git a/doc/release-notes/11752-croissant-restricted.md b/doc/release-notes/11752-croissant-restricted.md new file mode 100644 index 00000000000..ced0ef4eaa7 --- /dev/null +++ b/doc/release-notes/11752-croissant-restricted.md @@ -0,0 +1,13 @@ +- The optional Croissant exporter has been updated to 0.1.6 to prevent variable names, variable descriptions, and variable types from being exposed for restricted files. See https://github.com/gdcc/exporter-croissant/pull/20 and #11752. + +## Upgrade Instructions + +### Update Croissant exporter, if enabled, and reexport metadata + +If you have enabled the Croissant dataset metadata exporter, you should upgrade to version 0.1.6. + +- Stop Payara. +- Delete the old Croissant exporter jar file. It will be located in the directory defined by the `dataverse.spi.exporters.directory` setting. +- Download the updated Croissant jar from https://repo1.maven.org/maven2/io/gdcc/export/croissant/ and place it in the same directory. +- Restart Payara. +- Run reExportAll. diff --git a/doc/release-notes/11766-new-io.gdcc.dataverse-spi.md b/doc/release-notes/11766-new-io.gdcc.dataverse-spi.md new file mode 100644 index 00000000000..ef9e68f5719 --- /dev/null +++ b/doc/release-notes/11766-new-io.gdcc.dataverse-spi.md @@ -0,0 +1,2 @@ +The ExportDataProvider framework in the dataverse-spi package has been extended, adding some extra options for developers of metadata exporter plugins. +See the [documentation](https://guides.dataverse.org/en/latest/developers/metadataexport.html#building-an-exporter) in the Metadata Export guide for details. \ No newline at end of file diff --git a/doc/release-notes/11777-MDC-citation-api-improvement.md b/doc/release-notes/11777-MDC-citation-api-improvement.md new file mode 100644 index 00000000000..9441e9e0f44 --- /dev/null +++ b/doc/release-notes/11777-MDC-citation-api-improvement.md @@ -0,0 +1,7 @@ +The /api/admin/makeDataCount/{id}/updateCitationsForDataset endpoint, which allows citations for a dataset to be retrieved from DataCite, is often called periodically for all datasets. However, allowing calls for many datasets to be processed in parallel can cause performance problems in Dataverse and/or cause calls to DataCite to fail due to rate limiting. The existing implementation was also inefficient w.r.t. memory use when used on datasets with many (>~1K) files. This release configures Dataverse to queue calls to this api, processes them serially, adds optional throttling to avoid hitting DataCite rate limits and improves memory use. + +New optional MPConfig setting: + +dataverse.api.mdc.min-delay-ms - number of milliseconds to wait between calls to DataCite. A value of ~100 should conservatively address DataCite's current 3000/5 minute limit. A value of 250 may be required for their test service. + +Backward compatibility: This api call is now asynchronous and will return an OK response when the call is queued or a 503 if the queue is full. \ No newline at end of file diff --git a/doc/release-notes/11783-Curation-Status-fixes.md b/doc/release-notes/11783-Curation-Status-fixes.md new file mode 100644 index 00000000000..62b33165d0c --- /dev/null +++ b/doc/release-notes/11783-Curation-Status-fixes.md @@ -0,0 +1 @@ +In prior versions of Dataverse, publishing a dataset via the superuser only update-current-version option would not set the current curation status (if enabled/used) to none/empty and, in v6.7, would not maintain the curation status history. These issues are now resolved and the update current version option works the same as normal publication of a new version w.r.t. curation status. \ No newline at end of file diff --git a/doc/release-notes/11800-qb-ra-fixes.md b/doc/release-notes/11800-qb-ra-fixes.md new file mode 100644 index 00000000000..5c7ef82f7f2 --- /dev/null +++ b/doc/release-notes/11800-qb-ra-fixes.md @@ -0,0 +1,2 @@ +This release fixes problems with guestbook questions being displayed at download when files are selected from the dataset files table +when guestbook-at-request is enabled and not displaying when they should when access is requested from the file page. \ No newline at end of file diff --git a/doc/release-notes/11822-faster-permission-indexing.md b/doc/release-notes/11822-faster-permission-indexing.md new file mode 100644 index 00000000000..f716655232e --- /dev/null +++ b/doc/release-notes/11822-faster-permission-indexing.md @@ -0,0 +1 @@ +Permission reindexing, which occurs, e.g., after a user has been granted a role on a collection, has been made faster and less memory intensive in this release. \ No newline at end of file diff --git a/doc/sphinx-guides/source/admin/dataverses-datasets.rst b/doc/sphinx-guides/source/admin/dataverses-datasets.rst index a37819c90e1..0fa5bcf69f1 100644 --- a/doc/sphinx-guides/source/admin/dataverses-datasets.rst +++ b/doc/sphinx-guides/source/admin/dataverses-datasets.rst @@ -60,6 +60,10 @@ The current driver can be seen using:: curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver +Or to recurse the chain of parents to find the effective storageDriver:: + + curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver?getEffective=true + (Note that for ``dataverse.files.store1.label=MyLabel``, ``store1`` will be returned.) and can be reset to the default store with:: diff --git a/doc/sphinx-guides/source/admin/make-data-count.rst b/doc/sphinx-guides/source/admin/make-data-count.rst index 0103a6f9e38..f8ffa7bb084 100644 --- a/doc/sphinx-guides/source/admin/make-data-count.rst +++ b/doc/sphinx-guides/source/admin/make-data-count.rst @@ -166,6 +166,8 @@ The example :download:`counter_weekly.sh <../_static/util/counter_weekly.sh>` wi Citations will be retrieved for each published dataset and recorded in the your Dataverse installation's database. +Note that the :ref:`dataverse.api.mdc.min-delay-ms` setting can be used to avoid getting rate-limit errors from DataCite. + For how to get the citations out of your Dataverse installation, see "Retrieving Citations for a Dataset" under :ref:`Dataset Metrics ` in the :doc:`/api/native-api` section of the API Guide. Please note that while the Dataverse Software has a metadata field for "Related Dataset" this information is not currently sent as a citation to Crossref. diff --git a/doc/sphinx-guides/source/admin/monitoring.rst b/doc/sphinx-guides/source/admin/monitoring.rst index 16bb18b7ad2..ef4e4f4f206 100644 --- a/doc/sphinx-guides/source/admin/monitoring.rst +++ b/doc/sphinx-guides/source/admin/monitoring.rst @@ -149,7 +149,7 @@ Tips: - Use **Enhanced Monitoring**. Enhanced Monitoring gathers its metrics from an agent on the instance. See `Enhanced Monitoring docs `_. - It's possible to view and act on **RDS Events** such as snapshots, parameter changes, etc. See `Working with Amazon RDS events `_ for details. - RDS monitoring is available via API and the ``aws`` command line tool. For example, see `Retrieving metrics with the Performance Insights API `_. -- To play with monitoring RDS using a server configured by `dataverse-ansible `_ set ``use_rds`` to true to skip some steps that aren't necessary when using RDS. See also the :doc:`/developers/deployment` section of the Developer Guide. +- To play with monitoring RDS using a server configured by `dataverse-ansible `_ set ``use_rds`` to true to skip some steps that aren't necessary when using RDS. See also the :doc:`/developers/deployment` section of the Developer Guide. MicroProfile Metrics endpoint ----------------------------- diff --git a/doc/sphinx-guides/source/api/changelog.rst b/doc/sphinx-guides/source/api/changelog.rst index 5be6c78adce..d6523bfbdbc 100644 --- a/doc/sphinx-guides/source/api/changelog.rst +++ b/doc/sphinx-guides/source/api/changelog.rst @@ -7,12 +7,18 @@ This API changelog is experimental and we would love feedback on its usefulness. :local: :depth: 1 +v6.9 +---- +- The POST /api/admin/makeDataCount/{id}/updateCitationsForDataset processing is now asynchronous and the response no longer includes the number of citations. The response can be OK if the request is queued or 503 if the queue is full (default queue size is 1000). + v6.8 ---- - For POST /api/files/{id}/metadata passing an empty string ("description":"") or array ("categories":[]) will no longer be ignored. Empty fields will now clear out the values in the file's metadata. To ignore the fields simply do not include them in the JSON string. - For PUT /api/datasets/{id}/editMetadata the query parameter "sourceInternalVersionNumber" has been removed and replaced with "sourceLastUpdateTime" to verify that the data being edited hasn't been modified and isn't stale. - For GET /api/dataverses/$dataverse-alias/links the Json response has changed breaking the backward compatibility of the API. +- For GET /api/admin/dataverse/{dataverse-alias}/storageDriver and /api/datasets/{identifier}/storageDriver the driver name is no longer returned in data.message. This value is now returned in data.name. +- For PUT /api/dataverses/$dataverse-alias/inputLevels custom input levels that had been previously set will no longer be deleted. To delete input levels send an empty list (deletes all), then send the new/modified list. - For GET /api/externalTools and /api/externalTools/{id} the responses are now formatted as JSON (previously the toolParameters and allowedApiCalls were a JSON object and array (respectively) that were serialized as JSON strings) and any configured "requirements" are included. v6.7 diff --git a/doc/sphinx-guides/source/api/external-tools.rst b/doc/sphinx-guides/source/api/external-tools.rst index ae0e44b36aa..389519318db 100644 --- a/doc/sphinx-guides/source/api/external-tools.rst +++ b/doc/sphinx-guides/source/api/external-tools.rst @@ -202,7 +202,7 @@ Testing Your External Tool As the author of an external tool, you are not expected to learn how to install and operate a Dataverse installation. There's a very good chance your tool can be added to a server Dataverse Community developers use for testing if you reach out on any of the channels listed under :ref:`getting-help-developers` in the Developer Guide. -By all means, if you'd like to install a Dataverse installation yourself, a number of developer-centric options are available. For example, there's a script to spin up a Dataverse installation on EC2 at https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible . The process for using curl to add your external tool to your Dataverse installation is documented under :ref:`managing-external-tools` in the Admin Guide. +By all means, if you'd like to install a Dataverse installation yourself, a number of developer-centric options are available. For example, there's a script to spin up a Dataverse installation on EC2 at https://github.com/gdcc/dataverse-ansible . The process for using curl to add your external tool to your Dataverse installation is documented under :ref:`managing-external-tools` in the Admin Guide. Spreading the Word About Your External Tool ------------------------------------------- @@ -219,7 +219,7 @@ If you've thought to yourself that there ought to be an app store for Dataverse Demoing Your External Tool ++++++++++++++++++++++++++ -https://demo.dataverse.org is the place to play around with the Dataverse Software and your tool can be included. Please email support@dataverse.org to start the conversation about adding your tool. Additionally, you are welcome to open an issue at https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible which already includes a number of the tools listed above. +https://demo.dataverse.org is the place to play around with the Dataverse Software and your tool can be included. Please email support@dataverse.org to start the conversation about adding your tool. Additionally, you are welcome to open an issue at https://github.com/gdcc/dataverse-ansible which already includes a number of the tools listed above. Announcing Your External Tool +++++++++++++++++++++++++++++ diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index fa4b4611559..a12dfa151c9 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -732,6 +732,50 @@ Note: you must have "Add Dataset" permission in the given collection to invoke t .. _featured-collections: +List Dataverse Collections to Which a Given Dataset or Dataverse Collection May Be Linked +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The user may provide a search term to limit the list of Dataverse Collections returned. The search term will be compared to the name of the Dataverse Collections. +The response is a JSON array of the ids, aliases, and names of the Dataverse collections to which a given Dataset or Dataverse Collection may be linked: + +For a given Dataverse Collection: + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export OBJECT_TYPE=dataverse + export ID=collectionAlias + export SEARCH_TERM=searchOn + + curl -H "X-Dataverse-key:$API_TOKEN" -X GET "$SERVER_URL/api/dataverses/$ID/$OBJECT_TYPE/linkingDataverses?searchTerm=$SEARCH_TERM" + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/dataverses/collectionAlias/dataverse/linkingDataverses?searchTerm=searchOn" + +For a given Dataset: + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export OBJECT_TYPE=dataset + export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/J8SJZB + export SEARCH_TERM=searchOn + + curl -H "X-Dataverse-key:$API_TOKEN" -X GET "$SERVER_URL/api/dataverses/:persistentId/$OBJECT_TYPE/linkingDataverses?searchTerm=SEARCH_TERM&persistentId=$PERSISTENT_IDENTIFIER" + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/dataverses/:persistentId/dataset/linkingDataverses?searchTerm=searchOn&persistentId=doi:10.5072/FK2/J8SJZB" + +You may also add an optional "alreadyLinked=true" parameter to return collections which are already linked to the given Dataset or Dataverse Collection. + List Featured Collections for a Dataverse Collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1106,7 +1150,8 @@ Update Collection Input Levels Updates the dataset field type input levels in a collection. -Please note that this endpoint overwrites all the input levels of the collection page, so if you want to keep the existing ones, you will need to add them to the JSON request body. +Please note that this endpoint does not change previously updated input levels of the collection page, so if you want to add new levels or modify existing ones, you will need to include them in the JSON request body. +In order to delete input levels you must call this API with an empty list to delete all of the input levels, then call this API with the new list of input levels. If one of the input levels corresponds to a dataset field type belonging to a metadata block that does not exist in the collection, the metadata block will be added to the collection. diff --git a/doc/sphinx-guides/source/container/app-image.rst b/doc/sphinx-guides/source/container/app-image.rst index afffeae1c0b..47055bd3786 100644 --- a/doc/sphinx-guides/source/container/app-image.rst +++ b/doc/sphinx-guides/source/container/app-image.rst @@ -81,7 +81,7 @@ For now, stale images will be kept on Docker Hub indefinitely. | Example: :substitution-code:`|nextVersion|-noble` | Summary: Rolling tag, equivalent to ``unstable`` for current development cycle. Will roll over to the rolling production tag after a Dataverse release. - | Discussion: Perhaps you are eager to starting testing features of an upcoming version (e.g. |nextVersion|) in a staging environment. You select the :substitution-code:`|nextVersion|-noble` tag (as opposed to ``unstable``) because you want to stay on |nextVersion| rather switching to the version **after that** when a release is made (which would happen if you had selected the ``unstable`` tag). Also, when the next release comes out (|nextVersion| in this example), you would stay on the :substitution-code:`|nextVersion|-noble` tag, which is the same tag that someone would use who wants the final release of |nextVersion|. (See "Rolling Production", above.) + | Discussion: Perhaps you are eager to start testing features of an upcoming version (e.g. |nextVersion|) in a staging environment. You select the :substitution-code:`|nextVersion|-noble` tag (as opposed to ``unstable``) because you want to stay on |nextVersion| rather than switching to the version **after that** when a release is made (which would happen if you had selected the ``unstable`` tag). Also, when the next release comes out (|nextVersion| in this example), you would stay on the :substitution-code:`|nextVersion|-noble` tag, which is the same tag that someone would use who wants the final release of |nextVersion|. (See "Rolling Production", above.) **NOTE**: In these tags for development usage, the version number will always be 1 minor version ahead of existing Dataverse releases. Example: Assume Dataverse ``6.x`` is released, ``6.(x+1)`` is underway. diff --git a/doc/sphinx-guides/source/container/running/demo.rst b/doc/sphinx-guides/source/container/running/demo.rst index d4afee8a18a..32de0ea48bf 100644 --- a/doc/sphinx-guides/source/container/running/demo.rst +++ b/doc/sphinx-guides/source/container/running/demo.rst @@ -261,7 +261,7 @@ You should be able to see the new fields from the metadata block you added in th ``curl http://localhost:8983/solr/collection1/schema/fields`` -At this point you can proceed with testing the metadata block in the Dataverse UI. First you'll need to enable it for a collection (see :ref:`general-information` in the User Guide section about collection). Afterwards, create a new dataset, save it, and then edit the metadata for that dataset. Your metadata block should appear. +At this point you can proceed with testing the metadata block in the Dataverse UI. First you'll need to enable it for a collection (see :ref:`general-information` in the User Guide section about collections). Afterwards, create a new dataset, save it, and then edit the metadata for that dataset. Your metadata block should appear. Next Steps ---------- diff --git a/doc/sphinx-guides/source/contributor/documentation.md b/doc/sphinx-guides/source/contributor/documentation.md index 2a8d6794921..614dba1d6a0 100644 --- a/doc/sphinx-guides/source/contributor/documentation.md +++ b/doc/sphinx-guides/source/contributor/documentation.md @@ -50,76 +50,78 @@ If you would like to read more about the Dataverse's use of GitHub, please see t ## Building the Guides with Sphinx -While the "quick fix" technique shown above should work fine for minor changes, especially for larger changes, we recommend installing Sphinx on your computer or using a Sphinx Docker container to build the guides locally so you can get an accurate preview of your changes. +While the "quick fix" technique shown above should work fine for minor changes, in many cases, you're going to want to preview changes locally before committing them. -In case you decide to use a Sphinx Docker container to build the guides, you can skip the next two installation sections, but you will need to have Docker installed. +Before we worry about pushing changes to the code, let's make sure we can build the guides. -### Installing Sphinx +Go to and click "Code" and then follow the instructions to clone the code locally. -First, make a fork of and clone your fork locally. Then change to the ``doc/sphinx-guides`` directory. +### Docker -``cd doc/sphinx-guides`` +Install [Docker Desktop](https://www.docker.com/products/docker-desktop/). -Create a Python virtual environment, activate it, then install dependencies: +From a terminal, switch to the "dataverse" directory you just cloned. This is the root of the git repo. -``python3 -m venv venv`` +`cd dataverse` -``source venv/bin/activate`` +Then try running this command: -``pip install -r requirements.txt`` +`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx:7.2.6 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make html"` -### Installing GraphViz +If all goes well, you should be able to open `doc/sphinx-guides/build/html/index.html` to see the guides you just built. -In some parts of the documentation, graphs are rendered as images using the Sphinx GraphViz extension. +#### Docker with a Makefile -Building the guides requires the ``dot`` executable from GraphViz. +Once you've confirmed you have Docker working, if you have [make](https://en.wikipedia.org/wiki/Make_(software)) installed, you can try the following commands: -This requires having [GraphViz](https://graphviz.org) installed and either having ``dot`` on the path or -[adding options to the `make` call](https://groups.google.com/forum/#!topic/sphinx-users/yXgNey_0M3I). +`make docs-html` -On a Mac we recommend installing GraphViz through [Homebrew](). Once you have Homebrew installed and configured to work with your shell, you can type `brew install graphviz`. +`make docs-pdf` -### Editing and Building the Guides +`make docs-epub` -To edit the existing documentation: +`make docs-all` -- Create a branch (see {ref}`how-to-make-a-pull-request`). -- In ``doc/sphinx-guides/source`` you will find the .rst files that correspond to https://guides.dataverse.org. -- Using your preferred text editor, open and edit the necessary files, or create new ones. +### Sphinx Installed Locally -Once you are done, you can preview the changes by building the guides locally. As explained, you can build the guides with Sphinx locally installed, or with a Docker container. +First, run `python --version` or `python3 --version` to determine the version of Python you have. If you don't have Python 3.10 or higher, you must upgrade. -#### Building the Guides with Sphinx Installed Locally - -Open a terminal, change directories to `doc/sphinx-guides`, activate (or reactivate) your Python virtual environment, and build the guides. +Next, change to the `doc/sphinx-guides` directory. `cd doc/sphinx-guides` +Create a Python virtual environment, activate it, then install dependencies: + +`python3 -m venv venv` + `source venv/bin/activate` -`make clean` +`pip install -r requirements.txt` -`make html` +Next, install [GraphViz](https://graphviz.org) because building the guides requires having the `dot` executable from GraphViz either on the path or passed [as an argument](https://groups.google.com/g/sphinx-users/c/yXgNey_0M3I/m/3T2NipFlBgAJ). -#### Building the Guides with a Sphinx Docker Container and a Makefile +On a Mac we recommend installing GraphViz through [Homebrew](). Once you have Homebrew installed and configured to work with your shell, you can type `brew install graphviz`. -We have added a Makefile to simplify the process of building the guides using a Docker container, you can use the following commands from the repository root: +Finally, you can try building the guides with the following command. -- `make docs-html` -- `make docs-pdf` -- `make docs-epub` -- `make docs-all` +`make html` -#### Building the Guides with a Sphinx Docker Container and CLI +If all goes well, you should be able to open `doc/sphinx-guides/build/html/index.html` to see the guides you just built. -If you want to build the guides using a Docker container, execute the following command in the repository root: +## Editing, Building, and Previewing the Guides -`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx:7.2.6 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make html"` +To edit the existing documentation: -#### Previewing the Guides +- Create a branch (see {ref}`how-to-make-a-pull-request`). +- In `doc/sphinx-guides/source` you will find the .rst or .md files that correspond to https://guides.dataverse.org. +- Using your preferred text editor, open and edit the necessary files, or create new ones. + +Once you are done, you can preview the changes by building the guides using one of the options above. After Sphinx is done processing the files you should notice that the `html` folder in `doc/sphinx-guides/build` directory has been updated. You can click on the files in the `html` folder to preview the changes. +## Making a Pull Request + Now you can make a commit with the changes to your own fork in GitHub and submit a pull request. See {ref}`how-to-make-a-pull-request`. ## Writing Guidelines @@ -153,16 +155,22 @@ If the page is written in Markdown (.md), use this form: ### Links -Getting links right with .rst files can be tricky. +Getting links right can be tricky. #### Custom Titles -You can use a custom title when linking to a document like this: +In .rst files you can use a custom title when linking to a document like this: :doc:`Custom title ` See also +In .md files, the same pattern can be used. Here's an example of using a custom title with a ref: + + {ref}`Log in ` + +See also + ### Images A good documentation is just like a website enhanced and upgraded by adding high quality and self-explanatory images. Often images depict a lot of written text in a simple manner. Within our Sphinx docs, you can add them in two ways: a) add a PNG image directly and include or b) use inline description languages like GraphViz (current only option). diff --git a/doc/sphinx-guides/source/developers/deployment.rst b/doc/sphinx-guides/source/developers/deployment.rst index 46cf95dae54..ec9929136b7 100755 --- a/doc/sphinx-guides/source/developers/deployment.rst +++ b/doc/sphinx-guides/source/developers/deployment.rst @@ -78,7 +78,7 @@ Amazon offers instructions on using an IAM role to grant permissions to applicat Configure Ansible File (Optional) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In order to configure Dataverse installation settings such as the password of the dataverseAdmin user, download https://raw.githubusercontent.com/GlobalDataverseCommunityConsortium/dataverse-ansible/master/defaults/main.yml and edit the file to your liking. +In order to configure Dataverse installation settings such as the password of the dataverseAdmin user, download https://raw.githubusercontent.com/gdcc/dataverse-ansible/develop/defaults/main.yml and edit the file to your liking. You can skip this step if you're fine with the values in the "main.yml" file in the link above. @@ -89,7 +89,7 @@ Once you have done the configuration above, you are ready to try running the "ec Download `ec2-create-instance.sh`_ and put it somewhere reasonable. For the purpose of these instructions we'll assume it's in the "Downloads" directory in your home directory. -.. _ec2-create-instance.sh: https://raw.githubusercontent.com/GlobalDataverseCommunityConsortium/dataverse-ansible/master/ec2/ec2-create-instance.sh +.. _ec2-create-instance.sh: https://raw.githubusercontent.com/gdcc/dataverse-ansible/develop/ec2/ec2-create-instance.sh To run the script, you can make it executable (``chmod 755 ec2-create-instance.sh``) or run it with bash, like this with ``-h`` as an argument to print the help: diff --git a/doc/sphinx-guides/source/developers/making-library-releases.rst b/doc/sphinx-guides/source/developers/making-library-releases.rst index be867f9196a..0daa7fb89db 100755 --- a/doc/sphinx-guides/source/developers/making-library-releases.rst +++ b/doc/sphinx-guides/source/developers/making-library-releases.rst @@ -36,6 +36,32 @@ Releasing a Snapshot Version to Maven Central That is to say, to make a snapshot release, you only need to get one or more commits into the default branch. +It's possible, of course, to make snapshot releases outside of GitHub Actions, from environments such as your laptop. Generally, you'll want to look at the GitHub Action and try to do the equivalent. You'll need a file set up locally at ``~/.m2/settings.xml`` with the following (contact a core developer for the redacted bits): + +.. code-block:: bash + + + + + central + REDACTED + REDACTED + + + + +Then, study the GitHub Action and perform similar commands from your local environment. For example, as of this writing, for the dataverse-spi project, you can run the following commands, substituting the suffix you need: + +``mvn -f modules/dataverse-spi -Dproject.version.suffix="2.1.0-PR11767-SNAPSHOT" verify`` + +``mvn -f modules/dataverse-spi -Dproject.version.suffix="2.1.0-PR11767-SNAPSHOT" deploy`` + +This will upload the snapshot here, for example: https://central.sonatype.com/repository/maven-snapshots/io/gdcc/dataverse-spi/2.1.02.1.0-PR11767-SNAPSHOT/dataverse-spi-2.1.02.1.0-PR11767-20250827.182026-1.jar + +Before OSSRH was retired, you could browse through snapshot jars you published at https://s01.oss.sonatype.org/content/repositories/snapshots/io/gdcc/dataverse-spi/2.0.0-PR9685-SNAPSHOT/, for example. Now, even though you may see the URL of the jar as shown above during the "deploy" step, if you try to browse the various snapshot jars at https://central.sonatype.com/repository/maven-snapshots/io/gdcc/dataverse-spi/2.1.02.1.0-PR11767-SNAPSHOT/ you'll see "This maven2 hosted repository is not directly browseable at this URL. Please use the browse or HTML index views to inspect the contents of this repository." Sadly, the "browse" and "HTML index" links don't work, as noted in a `question `_ on the Sonatype Community forum. Below is a suggestion for confirming that the jar was uploaded properly, which is to use Maven to copy the jar to your local directory. You could then compare checksums. + +``mvn dependency:copy -DrepoUrl=https://central.sonatype.com/repository/maven-snapshots/ -Dartifact=io.gdcc:dataverse-spi:2.1.02.1.0-PR11767-SNAPSHOT -DoutputDirectory=.`` + Releasing a Release (Non-Snapshot) Version to Maven Central ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index 028b80e2892..fbbc2e5d3ae 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -8,9 +8,13 @@ Making Releases Introduction ------------ -This document is about releasing the main Dataverse app (https://github.com/IQSS/dataverse). See :doc:`making-library-releases` for how to release our various libraries. Other projects have their own release documentation. +.. note:: This document is about releasing the main Dataverse app (https://github.com/IQSS/dataverse). See :doc:`making-library-releases` for how to release our various libraries. Other projects have their own release documentation. -Below you'll see branches like "develop" and "master" mentioned. For more on our branching strategy, see :doc:`version-control`. +.. note:: Below you'll see branches like "develop" and "master" mentioned. For more on our branching strategy, see :doc:`version-control`. + +Dataverse releases are time-based as opposed to being feature-based. That is, we announce an approximate release date in advance (e.g. for `6.8 `_) and try to hit that deadline. If features we're working on aren't ready yet, the train will leave the station without them. We release quarterly. + +We also announce "last call" dates for both community pull requests and those made by core developers. If you are part of the community and have made a pull request, you have until this date to ask the team to add the upcoming milestone to your pull request. The same goes for core developers. This is not a guarantee that these pull requests will be reviewed, tested, QA'ed and merged before :ref:`code freeze `, but we'll try. Regular or Hotfix? ------------------ @@ -30,7 +34,9 @@ Early on, make sure it's clear what type of release this is. The steps below des Ensure Issues Have Been Created ------------------------------- -Some of the steps in this document are well-served by having their own dedicated GitHub issue. You'll see a label like this on them: +We have a "create release issues" script at https://github.com/IQSS/dv-project-metrics that should be run a week or so before code freeze. + +For each issue that is created by the script there is likely a corresponding step in this document that has "dedicated" label on it like this: |dedicated| @@ -41,35 +47,39 @@ There are a variety of reasons why a step might deserve its own dedicated issue: Steps don't get their own dedicated issue if it would be confusing to have multiple people involved. Too many cooks in the kitchen, as they say. Also, some steps are so small the overhead of an issue isn't worth it. -Before the release even begins you can coordinate with the project manager about the creation of these issues. - .. |dedicated| raw:: html Dedicated Issue   +.. _declare-code-freeze: + Declare a Code Freeze --------------------- -The following steps are made more difficult if code is changing in the "develop" branch. Declare a code freeze until the release is out. Do not allow pull requests to be merged. +When we declare a code freeze, we mean: -For a hotfix, a code freeze (no merging) is necessary not because we want code to stop changing in the branch being hotfix released, but because bumping the version used in Jenkins/Ansible means that API tests will fail in pull requests until the version is bumped in those pull requests. +- No additional features will be merged until the freeze is lifted. +- Bug fixes will only be merged if they relate to the upcoming release in some way, such as fixes for regressions or performance problems in that release. +- Pull requests that directly affect the release, such as bumping the version, will be merged, of course. -Conduct Performance Testing ---------------------------- +The benefits of the code freeze are: -|dedicated| +- The team can focus on getting the release out together. +- Regression and performance testing can happen on code that isn't changing. +- The release notes can be written without having to worry about new features (and their release note snippets) being merged in. -See :doc:`/qa/performance-tests` for details. +In short, the steps described below become easier under a code freeze. -Conduct Regression Testing ---------------------------- +Note: for a hotfix, a code freeze is necessary not because we want code to stop changing in the branch being hotfix released, but because bumping the version used in Jenkins/Ansible means that API tests will fail in pull requests until the version is bumped in those pull requests. Basically, we want to get the hotfix merged quickly so we can propagate the version bump into all open pull requests so that API tests can start passing again in those pull requests. -|dedicated| +Push Back Milestones on Pull Requests That Missed the Train +----------------------------------------------------------- -See :doc:`/qa/testing-approach` for details. -Refer to the provided regression checklist for the list of items to verify during the testing process: `Regression Checklist `_. +As of this writing, we optimistically add milestones to issues and pull requests, hoping that the work will be complete before code freeze. Inevitably, we're a bit too optimistic. + +Hopefully, as the release approached, the team has already decided which pull requests (that aren't related to the release) won't make the cut. If not, go ahead and bump them to the next release. .. _write-release-notes: @@ -85,7 +95,7 @@ The task at or near release time is to collect these snippets into a single file - Find the issue in GitHub that tracks the work of creating release notes for the upcoming release. - Create a branch, add a .md file for the release (ex. 5.10.1 Release Notes) in ``/doc/release-notes`` and write the release notes, making sure to pull content from the release note snippets mentioned above. Snippets may not include any issue number or pull request number in the text so be sure to copy the number from the filename of the snippet into the final release note. - Delete (``git rm``) the release note snippets as the content is added to the main release notes file. -- Include instructions describing the steps required to upgrade the application from the previous version. These must be customized for release numbers and special circumstances such as changes to metadata blocks and infrastructure. +- Include instructions describing the steps required to upgrade the application from the previous version. These must be customized for release numbers and special circumstances such as changes to metadata blocks and infrastructure. These instructions are required for the next steps (deploying to various environments) so try to prioritize them over finding just the right words in release highlights (which you can do later). - Make a pull request. Here's an example: https://github.com/IQSS/dataverse/pull/11613 - Note that we won't merge the release notes until after we have confirmed that the upgrade instructions are valid by performing a couple upgrades. @@ -110,12 +120,58 @@ ssh into the dataverse-internal server and download the release candidate war fi Go to /doc/release-notes, open the release-notes.md file for the release we're working on, and perform all the steps under "Upgrade Instructions". Note that for regular releases, we haven't bumped the version yet so you won't be able to follow the steps exactly. (For hotfix releases, the version will be bumped already.) +Deploy Release Candidate to QA +------------------------------ + +|dedicated| + +Deploy the same war file to https://qa.dataverse.org using the same upgrade instructions as above. + +Solicit Feedback from Curation Team +----------------------------------- + +Ask the curation team to test on https://qa.dataverse.org and give them five days to provide feedback. + + +Conduct Performance Testing +--------------------------- + +|dedicated| + +See :doc:`/qa/performance-tests` for details. + +Conduct Regression Testing +--------------------------- + +|dedicated| + +Regression testing should be conducted on production data. +See :doc:`/qa/testing-approach` for details. +Refer to the provided regression checklist for the list of items to verify during the testing process: `Regression Checklist `_. + +Build the Guides for the Release Candidate +------------------------------------------ + +Go to https://jenkins.dataverse.org/job/guides.dataverse.org/ and make the following adjustments to the config: + +- Repository URL: ``https://github.com/IQSS/dataverse.git`` +- Branch Specifier (blank for 'any'): ``*/develop`` +- ``VERSION`` (under "Build Steps"): use the next release version but add "-rc.1" to the end. Don't prepend a "v". Use ``6.8-rc.1`` (for example) + +Click "Save" then "Build Now". + +Make sure the guides directory appears in the expected location such as https://guides.dataverse.org/en/6.8-rc.1/ + +When previewing the HTML version of docs from pull requests, we don't usually use this Jenkins job, relying instead on automated ReadTheDocs builds. The reason for doing this step now while we wait for feedback from the Curation Team is that it's an excellent time to fix the Jenkins job, if necessary, to accommodate any changes needed to continue to build the docs. For example, Sphinx might need to be updated or a dependency might need to be installed. Such changes should be listed in the release notes for documentation writers. + Deploy Release Candidate to Demo -------------------------------- |dedicated| -Deploy the same war file to https://demo.dataverse.org using the same upgrade instructions as above. +Time has passed. The curation team has given feedback. We've finished regression and performance testing. Fixes may have been merged into the "develop" branch. We're ready to actually make the release now, which includes deploying a release candidate to the demo server. + +Build a new war file, if necessary, and deploy it to https://demo.dataverse.org using the upgrade instructions in the release notes. Merge Release Notes (Once Ready) -------------------------------- @@ -171,16 +227,25 @@ Merge "develop" into "master" (non-hotfix only) If this is a regular (non-hotfix) release, create a pull request to merge the "develop" branch into the "master" branch using this "compare" link: https://github.com/IQSS/dataverse/compare/master...develop -Once important tests have passed (compile, unit tests, etc.), merge the pull request (skipping code review is ok). Don't worry about style tests failing such as for shell scripts. +Allow time for important tests (compile, unit tests, etc.) to pass. Don't worry about style tests failing such as for shell scripts. It's ok to skip code review. + +When merging the pull request, be sure to choose "create a merge commit" and not "squash and merge" or "rebase and merge". We suspect that choosing squash or rebase may have led to `lots of merge conflicts `_ when we tried to perform this "merge develop to master" step, forcing us to `re-do `_ the previous release before we could proceed with the current release. If this is a hotfix release, skip this whole "merge develop to master" step (the "develop" branch is not involved until later). +Confirm "master" Mergeability +----------------------------- + +Hopefully, the previous step went ok. As a sanity check, use the "compare" link at https://github.com/IQSS/dataverse/compare/master...develop again to look for merge conflicts without making a pull request. + +If the GitHub UI tells you there would be merge conflicts, something has gone horribly wrong (again) with the "merge develop to master" step. Stop and ask for help. + Add Milestone to Pull Requests and Issues ----------------------------------------- Often someone is making sure that the proper milestone (e.g. 5.10.1) is being applied to pull requests and issues, but sometimes this falls between the cracks. -Check for merged pull requests that have no milestone by going to https://github.com/IQSS/dataverse/pulls and entering `is:pr is:merged no:milestone `_ as a query. If you find any, add the milestone to the pull request and any issues it closes. This includes the "merge develop into master" pull request above. +Check for merged pull requests that have no milestone by going to https://github.com/IQSS/dataverse/pulls and entering `is:pr is:merged no:milestone `_ as a query. If you find any, first check if those pull requests are against open pull requests. If so, do nothing. Otherwise, add the milestone to the pull request and any issues it closes. This includes the "merge develop into master" pull request above. (Optional) Test Docker Images ----------------------------- diff --git a/doc/sphinx-guides/source/developers/testing.rst b/doc/sphinx-guides/source/developers/testing.rst index 1690864d453..f84a7cf1ac7 100755 --- a/doc/sphinx-guides/source/developers/testing.rst +++ b/doc/sphinx-guides/source/developers/testing.rst @@ -128,7 +128,7 @@ You might find studying the following test classes helpful in writing tests for - DeletePrivateUrlCommandTest.java - GetPrivateUrlCommandTest.java -In addition, there is a writeup on "The Testable Command" at https://github.com/IQSS/dataverse/blob/develop/doc/theTestableCommand/TheTestableCommand.md . +In addition, there is a writeup on "The Testable Command" at https://github.com/IQSS/dataverse/blob/master/doc/theTestableCommand/TheTestableCommand.md . Running Non-Essential (Excluded) Unit Tests ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -169,12 +169,12 @@ different people. For our purposes, an integration test can have two flavors: Running the Full API Test Suite Using EC2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**Prerequisite:** To run the API test suite in an EC2 instance you should first follow the steps in the :doc:`deployment` section to get set up with the AWS binary to launch EC2 instances. If you're here because you just want to spin up a branch, you'll still want to follow the AWS deployment setup steps, but may find the `ec2-create README.md `_ Quick Start section helpful. +**Prerequisite:** To run the API test suite in an EC2 instance you should first follow the steps in the :doc:`deployment` section to get set up with the AWS binary to launch EC2 instances. If you're here because you just want to spin up a branch, you'll still want to follow the AWS deployment setup steps, but may find the `ec2-create README.md `_ Quick Start section helpful. -You may always retrieve a current copy of the ec2-create-instance.sh script and accompanying group_var.yml file from the `dataverse-ansible repo `_. Since we want to run the test suite, let's grab the group_vars used by Jenkins: +You may always retrieve a current copy of the ec2-create-instance.sh script and accompanying group_var.yml file from the `dataverse-ansible repo `_. Since we want to run the test suite, let's grab the group_vars used by Jenkins: -- `ec2-create-instance.sh `_ -- `jenkins.yml `_ +- `ec2-create-instance.sh `_ +- `jenkins.yml `_ Edit ``jenkins.yml`` to set the desired GitHub repo and branch, and to adjust any other options to meet your needs: @@ -184,7 +184,7 @@ Edit ``jenkins.yml`` to set the desired GitHub repo and branch, and to adjust an - ``dataverse.unittests.enabled: true`` - ``dataverse.sampledata.enabled: true`` -If you wish, you may pass the script a ``-l`` flag with a local relative path in which the script will `copy various logs `_ at the end of the test suite for your review. +If you wish, you may pass the script a ``-l`` flag with a local relative path in which the script will `copy various logs `_ at the end of the test suite for your review. Finally, run the script: @@ -526,7 +526,7 @@ Browser-Based Testing Installation Testing ~~~~~~~~~~~~~~~~~~~~ -- Work with @donsizemore to automate testing of https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible +- Work with @donsizemore to automate testing of https://github.com/gdcc/dataverse-ansible Future Work on Load/Performance Testing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -538,4 +538,4 @@ Future Work on Load/Performance Testing Future Work on Accessibility Testing ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Using https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible and hooks available from accessibility testing tools, automate the running of accessibility tools on PRs so that developers will receive quicker feedback on proposed code changes that reduce the accessibility of the application. +- Using https://github.com/gdcc/dataverse-ansible and hooks available from accessibility testing tools, automate the running of accessibility tools on PRs so that developers will receive quicker feedback on proposed code changes that reduce the accessibility of the application. diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 14bf33c9482..171320919d9 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -3189,7 +3189,7 @@ dataverse.person-or-org.org-phrase-array Please note that this setting is experimental. The Schema.org metadata and OpenAIRE exports and the Schema.org metadata included in DatasetPages try to infer whether each entry in the various fields (e.g. Author, Contributor) is a Person or Organization. -If you have examples where an orgization name is being inferred to belong to a person, you can use this setting to force it to be recognized as an organization. +If you have examples where an organization name is being inferred to belong to a person, you can use this setting to force it to be recognized as an organization. The value is expected to be a comma-separated list of strings. Any name that contains one of the strings is assumed to be an organization. For example, "Project" is a word that is not otherwise associated with being an organization. Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_PERSON_OR_ORG_ORG_PHRASE_ARRAY``. @@ -3731,6 +3731,22 @@ Example: Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_CORS_HEADERS_EXPOSE``. + +.. _dataverse.api.mdc.min-delay-ms: + +dataverse.api.mdc.min-delay-ms +++++++++++++++++++++++++++++++ + +Minimum delay in milliseconds between Make Data Count (MDC) API requests from the /api/admin/makeDataCount/{id}/updateCitationsForDataset api. +This setting helps prevent overloading the MDC service by enforcing a minimum time interval between consecutive requests. +If a request arrives before this interval has elapsed since the previous request, it will be rate-limited. + +Default: ``0`` (no delay enforced) + +Example: ``dataverse.api.mdc.min-delay-ms=100`` (enforces a minimum 100ms delay between MDC API requests) + +Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_API_MDC_MIN_DELAY_MS``. + .. _feature-flags: Feature Flags diff --git a/doc/sphinx-guides/source/installation/prep.rst b/doc/sphinx-guides/source/installation/prep.rst index abb4349d3ad..33b1aa3c7b5 100644 --- a/doc/sphinx-guides/source/installation/prep.rst +++ b/doc/sphinx-guides/source/installation/prep.rst @@ -26,7 +26,7 @@ Advanced Installation There are some community-lead projects to use configuration management tools such as Ansible and Puppet to automate the installation and configuration of the Dataverse Software, but support for these solutions is limited to what the Dataverse Community can offer as described in each project's webpage: -- https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible +- https://github.com/gdcc/dataverse-ansible - https://gitlab.com/lip-computing/dataverse - https://github.com/IQSS/dataverse-puppet diff --git a/doc/sphinx-guides/source/qa/testing-approach.md b/doc/sphinx-guides/source/qa/testing-approach.md index 817161d02a0..49b9075cf7f 100644 --- a/doc/sphinx-guides/source/qa/testing-approach.md +++ b/doc/sphinx-guides/source/qa/testing-approach.md @@ -34,7 +34,7 @@ Think about risk. Is the feature or function part of a critical area such as per ## Smoke Test -1. Go to the homepage on . Scroll to the bottom to ensure the build number is the one you intend to test from Jenkins. +1. Go to the homepage on (this server has production data). Scroll to the bottom to ensure the build number is the one you intend to test from Jenkins. 1. Create a new user: It's fine to use a formulaic name with your initials and date and make the username and password the same, eg. kc080622. 1. Create a dataverse: You can use the same username. 1. Create a dataset: You can use the same username; fill in the required fields (do not use a template). diff --git a/makefile b/makefile index 315ff9c508c..1ffdb627275 100644 --- a/makefile +++ b/makefile @@ -1,5 +1,5 @@ - -SPHINX_VERSION = $(shell grep "Sphinx" ./doc/sphinx-guides/requirements.txt | awk -F'==' '{print $$2}') +# We use "Sphinx=" to avoid packages like Sphinx-Substitution-Extensions +SPHINX_VERSION = $(shell grep "Sphinx=" ./doc/sphinx-guides/requirements.txt | awk -F'==' '{print $$2}') docs-html: docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make clean && make html" @@ -11,4 +11,4 @@ docs-epub: docs-all: docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make clean && make html && make epub" - docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx-latexpdf:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make latexpdf LATEXMKOPTS=\"-interaction=nonstopmode\"; cd ../.. && ls -1 doc/sphinx-guides/build/latex/Dataverse.pdf" \ No newline at end of file + docker run -it --rm -v $$(pwd):/docs sphinxdoc/sphinx-latexpdf:$(SPHINX_VERSION) bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make latexpdf LATEXMKOPTS=\"-interaction=nonstopmode\"; cd ../.. && ls -1 doc/sphinx-guides/build/latex/Dataverse.pdf" diff --git a/modules/container-base/README.md b/modules/container-base/README.md index f6854482073..7a39457b766 100644 --- a/modules/container-base/README.md +++ b/modules/container-base/README.md @@ -25,7 +25,7 @@ provides in-depth information about content, building, tuning and so on for this **Where to get help and ask questions:** IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it. -You can join the Community Chat on Matrix at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community +You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community to ask for help and guidance. ## Supported Image Tags diff --git a/modules/container-configbaker/README.md b/modules/container-configbaker/README.md index 75862ee0809..8b68e3db692 100644 --- a/modules/container-configbaker/README.md +++ b/modules/container-configbaker/README.md @@ -19,7 +19,7 @@ provides information about this image. **Where to get help and ask questions:** IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it. -You can join the Community Chat on Matrix at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community +You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community to ask for help and guidance. ## Supported Image Tags diff --git a/modules/dataverse-spi/pom.xml b/modules/dataverse-spi/pom.xml index b00053fe5e0..a603e274234 100644 --- a/modules/dataverse-spi/pom.xml +++ b/modules/dataverse-spi/pom.xml @@ -13,7 +13,7 @@ io.gdcc dataverse-spi - 2.0.0${project.version.suffix} + 2.1.0${project.version.suffix} jar Dataverse SPI Plugin API @@ -64,11 +64,13 @@ - ossrh - https://s01.oss.sonatype.org/content/repositories/snapshots + central + https://central.sonatype.com/repository/maven-snapshots/ + ossrh + https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/ @@ -110,7 +112,9 @@ nexus-staging-maven-plugin true + ossrh + https://s01.oss.sonatype.org true diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java new file mode 100644 index 00000000000..9478d39c4c2 --- /dev/null +++ b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataContext.java @@ -0,0 +1,61 @@ +package io.gdcc.spi.export; + +/** + * + * @author landreev + * Provides an optional mechanism for defining various data retrieval options + * for the export subsystem in a way that should allow us adding support for + * more options going forward with minimal or no changes to the already + * implemented export plugins. + */ +public class ExportDataContext { + private boolean datasetMetadataOnly = false; + private boolean publicFilesOnly = false; + private Integer offset = null; + private Integer length = null; + + private ExportDataContext() { + + } + + public static ExportDataContext context() { + ExportDataContext context = new ExportDataContext(); + return context; + } + + public ExportDataContext withDatasetMetadataOnly() { + this.datasetMetadataOnly = true; + return this; + } + + public ExportDataContext withPublicFilesOnly() { + this.publicFilesOnly = true; + return this; + } + + public ExportDataContext withOffset(Integer offset) { + this.offset = offset; + return this; + } + + public ExportDataContext withLength(Integer length) { + this.length = length; + return this; + } + + public boolean isDatasetMetadataOnly() { + return datasetMetadataOnly; + } + + public boolean isPublicFilesOnly() { + return publicFilesOnly; + } + + public Integer getOffset() { + return offset; + } + + public Integer getLength() { + return length; + } +} diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java new file mode 100644 index 00000000000..69f813f83ce --- /dev/null +++ b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataOption.java @@ -0,0 +1,51 @@ +package io.gdcc.spi.export; + +/** + * + * @author landreev + * Provides a mechanism for defining various data retrieval options for the + * export subsystem in a way that should allow us adding support for more + * options going forward with minimal or no changes to the existing code in + * export plugins. + */ +@Deprecated +public class ExportDataOption { + + public enum SupportedOptions { + DatasetMetadataOnly, + PublicFilesOnly; + } + + private SupportedOptions optionType; + + /*public static ExportDataOption addOption(String option) { + ExportDataOption ret = new ExportDataOption(); + + for (SupportedOptions supported : SupportedOptions.values()) { + if (supported.toString().equals(option)) { + ret.optionType = supported; + } + } + return ret; + }*/ + + public static ExportDataOption addDatasetMetadataOnly() { + ExportDataOption ret = new ExportDataOption(); + ret.optionType = SupportedOptions.DatasetMetadataOnly; + return ret; + } + + public static ExportDataOption addPublicFilesOnly() { + ExportDataOption ret = new ExportDataOption(); + ret.optionType = SupportedOptions.PublicFilesOnly; + return ret; + } + + public boolean isDatasetMetadataOnly() { + return SupportedOptions.DatasetMetadataOnly.equals(optionType); + } + + public boolean isPublicFilesOnly() { + return SupportedOptions.PublicFilesOnly.equals(optionType); + } +} diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java index d039ac39e8f..4197d978e79 100644 --- a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java +++ b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/ExportDataProvider.java @@ -21,8 +21,14 @@ public interface ExportDataProvider { * OAI_ORE export are the only two that provide 'complete' * dataset-level metadata along with basic file metadata for each file * in the dataset. + * @param context - supplies optional parameters. Needs to support + * context.isDatasetMetadataOnly(). In a situation where we + * need to generate a format like DC that has no use for the + * file-level metadata, it makes sense to skip retrieving and + * formatting it, since there can be a very large number of + * files in a dataset. */ - JsonObject getDatasetJson(); + JsonObject getDatasetJson(ExportDataContext... context); /** * @@ -32,14 +38,15 @@ public interface ExportDataProvider { * @apiNote - THis, and the JSON format are the only two that provide complete * dataset-level metadata along with basic file metadata for each file * in the dataset. + * @param context - supplies optional parameters. */ - JsonObject getDatasetORE(); + JsonObject getDatasetORE(ExportDataContext... context); /** * Dataverse is capable of extracting DDI-centric metadata from tabular * datafiles. This detailed metadata, which is only available for successfully * "ingested" tabular files, is not included in the output of any other methods - * in this interface. + * in this interface. * * @return - a JSONArray with one entry per ingested tabular dataset file. * @apiNote - there is no JSON schema available for this output and the format @@ -47,9 +54,26 @@ public interface ExportDataProvider { * edu.harvard.iq.dataverse.export.DDIExporter and the @see * edu.harvard.iq.dataverse.util.json.JSONPrinter classes where this * output is used/generated (respectively). + * @param context - supplies optional parameters. */ - JsonArray getDatasetFileDetails(); + JsonArray getDatasetFileDetails(ExportDataContext... context); + /** + * Similar to the above, but + * a) retrieves the information for the ingested/tabular data files _only_ + * b) provides an option for retrieving this stuff in batches + * c) provides an option for skipping restricted/embargoed etc. files. + * Intended for datasets with massive numbers of tabular files and datavariables. + * @param context - supplies optional parameters. + * current (2.1.0) known use cases: + * context.isPublicFilesOnly(); + * context.getOffset(); + * context.getLength(); + * @return json array containing the datafile/filemetadata->datatable->datavariable metadata + * @throws ExportException + */ + JsonArray getTabularDataDetails(ExportDataContext ... context) throws ExportException; + /** * * @return - the subset of metadata conforming to the schema.org standard as @@ -58,8 +82,9 @@ public interface ExportDataProvider { * @apiNote - as this metadata export is not complete, it should only be used as * a starting point for an Exporter if it simplifies your exporter * relative to using the JSON or OAI_ORE exports. + * @param context - supplies optional parameters. */ - JsonObject getDatasetSchemaDotOrg(); + JsonObject getDatasetSchemaDotOrg(ExportDataContext... context); /** * @@ -68,8 +93,9 @@ public interface ExportDataProvider { * @apiNote - as this metadata export is not complete, it should only be used as * a starting point for an Exporter if it simplifies your exporter * relative to using the JSON or OAI_ORE exports. + * @param context - supplies optional parameters. */ - String getDataCiteXml(); + String getDataCiteXml(ExportDataContext... context); /** * If an Exporter has specified a prerequisite format name via the @@ -88,9 +114,10 @@ public interface ExportDataProvider { * malfunction, e.g. if you depend on format "ddi" and a third party * Exporter is configured to replace the internal ddi Exporter in * Dataverse. + * @param context - supplies optional parameters. */ - default Optional getPrerequisiteInputStream() { + default Optional getPrerequisiteInputStream(ExportDataContext... context) { return Optional.empty(); } - -} + + } diff --git a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java index 1338a3c9734..7132e74641b 100644 --- a/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java +++ b/modules/dataverse-spi/src/main/java/io/gdcc/spi/export/Exporter.java @@ -85,7 +85,6 @@ default Optional getPrerequisiteFormatName() { return Optional.empty(); } - /** * Harvestable Exporters will be available as options in Dataverse's Harvesting mechanism. * @return true to make this exporter available as a harvesting option. diff --git a/src/main/docker/README.md b/src/main/docker/README.md index 48416c196ca..a32c91a810e 100644 --- a/src/main/docker/README.md +++ b/src/main/docker/README.md @@ -24,7 +24,7 @@ for more details on tunable settings, locations, etc. **Where to get help and ask questions:** IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it. -You can join the Community Chat on Matrix at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community +You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community to ask for help and guidance. ## Supported Image Tags diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java index dec07a09643..28b061ffa2a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingDataverse.java @@ -9,6 +9,7 @@ import jakarta.persistence.Id; import jakarta.persistence.Index; import jakarta.persistence.JoinColumn; +import jakarta.persistence.NamedNativeQuery; import jakarta.persistence.NamedQueries; import jakarta.persistence.NamedQuery; import jakarta.persistence.OneToOne; @@ -35,6 +36,18 @@ @NamedQuery(name = "DatasetLinkingDataverse.findIdsByLinkingDataverseId", query = "SELECT o.dataset.id FROM DatasetLinkingDataverse AS o WHERE o.linkingDataverse.id = :linkingDataverseId") }) + + @NamedNativeQuery( + name = "DatasetLinkingDataverse.findByDatasetIdAndLinkingDataverseName", + query = """ + select o.linkingDataverse_id from DatasetLinkingDataverse as o + LEFT JOIN dataverse dv ON dv.id = o.linkingDataverse_id + WHERE o.dataset_id =? AND ((LOWER(dv.name) LIKE ? and ((SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?) + or (SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?))) + or (LOWER(dv.name) NOT LIKE ? and ((LOWER(dv.name) LIKE ?) + or (LOWER(dv.name) LIKE ?))))""" + ) + public class DatasetLinkingDataverse implements Serializable { private static final long serialVersionUID = 1L; @Id diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java index 39c82bfa3f1..6b0f8af6590 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetLinkingServiceBean.java @@ -5,6 +5,7 @@ */ package edu.harvard.iq.dataverse; +import jakarta.ejb.EJB; import java.util.ArrayList; import java.util.List; import java.util.logging.Logger; @@ -13,7 +14,6 @@ import jakarta.persistence.EntityManager; import jakarta.persistence.NoResultException; import jakarta.persistence.PersistenceContext; -import jakarta.persistence.Query; import jakarta.persistence.TypedQuery; /** @@ -28,6 +28,9 @@ public class DatasetLinkingServiceBean implements java.io.Serializable { @PersistenceContext(unitName = "VDCNet-ejbPU") private EntityManager em; + @EJB + DataverseServiceBean dataverseService; + public List findLinkedDatasets(Long dataverseId) { @@ -41,13 +44,42 @@ public List findLinkedDatasets(Long dataverseId) { } public List findLinkingDataverses(Long datasetId) { + return findLinkingDataverses(datasetId, ""); + } + + public List findLinkingDataverses(Long datasetId, String searchTerm) { List retList = new ArrayList<>(); - TypedQuery typedQuery = em.createNamedQuery("DatasetLinkingDataverse.findByDatasetId", DatasetLinkingDataverse.class) - .setParameter("datasetId", datasetId); - for (DatasetLinkingDataverse datasetLinkingDataverse : typedQuery.getResultList()) { - retList.add(datasetLinkingDataverse.getLinkingDataverse()); + if (searchTerm == null || searchTerm.isEmpty()) { + TypedQuery typedQuery = em.createNamedQuery("DatasetLinkingDataverse.findByDatasetId", DatasetLinkingDataverse.class) + .setParameter("datasetId", datasetId); + for (DatasetLinkingDataverse datasetLinkingDataverse : typedQuery.getResultList()) { + retList.add(datasetLinkingDataverse.getLinkingDataverse()); + } + return retList; + + } else { + + String pattern = searchTerm.toLowerCase(); + + String pattern1 = pattern + "%"; + String pattern2 = "% " + pattern + "%"; + + // Adjust the queries for very short, 1 and 2-character patterns: + if (pattern.length() == 1) { + pattern1 = pattern; + pattern2 = pattern + " %"; + } + TypedQuery typedQuery + = em.createNamedQuery("DatasetLinkingDataverse.findByDatasetIdAndLinkingDataverseName", Long.class) + .setParameter(1, datasetId).setParameter(2, "%dataverse").setParameter(3, pattern1) + .setParameter(4, pattern2).setParameter(5, "%dataverse").setParameter(6, pattern1).setParameter(7, pattern2); + + for (Long id : typedQuery.getResultList()) { + retList.add(dataverseService.find(id)); + } + return retList; } - return retList; + } public void save(DatasetLinkingDataverse datasetLinkingDataverse) { diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java index 3030922ea5e..bf2326f0e06 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingDataverse.java @@ -13,6 +13,7 @@ import jakarta.persistence.Id; import jakarta.persistence.Index; import jakarta.persistence.JoinColumn; +import jakarta.persistence.NamedNativeQuery; import jakarta.persistence.NamedQueries; import jakarta.persistence.NamedQuery; import jakarta.persistence.OneToOne; @@ -39,6 +40,16 @@ @NamedQuery(name = "DataverseLinkingDataverse.findIdsByLinkingDataverseId", query = "SELECT o.dataverse.id FROM DataverseLinkingDataverse AS o WHERE o.linkingDataverse.id = :linkingDataverseId") }) + @NamedNativeQuery( + name = "DataverseLinkingDataverse.findByDataverseIdAndLinkingDataverseName", + query = """ + select o.linkingDataverse_id from DataverseLinkingDataverse as o + LEFT JOIN dataverse dv ON dv.id = o.linkingDataverse_id + WHERE o.dataverse_id =? AND ((LOWER(dv.name) LIKE ? and ((SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?) + or (SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?))) + or (LOWER(dv.name) NOT LIKE ? and ((LOWER(dv.name) LIKE ?) + or (LOWER(dv.name) LIKE ?))))""" + ) public class DataverseLinkingDataverse implements Serializable { private static final long serialVersionUID = 1L; @Id diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java index 834ff96e392..9f1bcde4c0e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseLinkingServiceBean.java @@ -43,12 +43,41 @@ public List findLinkedDataverses(Long linkingDataverseId) { } public List findLinkingDataverses(Long dataverseId) { + + return findLinkingDataverses(dataverseId, ""); + } + + public List findLinkingDataverses(Long dataverseId, String searchTerm) { List retList = new ArrayList<>(); - TypedQuery typedQuery = em.createNamedQuery("DataverseLinkingDataverse.findByDataverseId", DataverseLinkingDataverse.class) - .setParameter("dataverseId", dataverseId); - for (DataverseLinkingDataverse dataverseLinkingDataverse : typedQuery.getResultList()) { - retList.add(dataverseLinkingDataverse.getLinkingDataverse()); + if (searchTerm == null || searchTerm.isEmpty()) { + TypedQuery typedQuery = em.createNamedQuery("DataverseLinkingDataverse.findByDataverseId", DataverseLinkingDataverse.class) + .setParameter("dataverseId", dataverseId); + for (DataverseLinkingDataverse dataverseLinkingDataverse : typedQuery.getResultList()) { + retList.add(dataverseLinkingDataverse.getLinkingDataverse()); + } + + } else { + + String pattern = searchTerm.toLowerCase(); + + String pattern1 = pattern + "%"; + String pattern2 = "% " + pattern + "%"; + + // Adjust the queries for very short, 1 and 2-character patterns: + if (pattern.length() == 1) { + pattern1 = pattern; + pattern2 = pattern + " %"; + } + TypedQuery typedQuery + = em.createNamedQuery("DataverseLinkingDataverse.findByDataverseIdAndLinkingDataverseName", Long.class) + .setParameter(1, dataverseId).setParameter(2, "%dataverse").setParameter(3, pattern1) + .setParameter(4, pattern2).setParameter(5, "%dataverse").setParameter(6, pattern1).setParameter(7, pattern2); + + for (Long id : typedQuery.getResultList()) { + retList.add(dataverseService.find(id)); + } } + return retList; } diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java index c14711060af..57bc0bc5450 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java @@ -505,17 +505,28 @@ public List filterByAliasQuery(String filterQuery) { return ret; } - public List filterDataversesForLinking(String query, DataverseRequest req, Dataset dataset) { + public List filterDataversesForLinking(String query, DataverseRequest req, DvObject dvo) { List dataverseList = new ArrayList<>(); List results = filterDataversesByNamePattern(query); - - if (results == null || results.size() == 0) { - return null; + + if (results == null || results.isEmpty()) { + return null; + } + + Dataset linkedDataset = null; + Dataverse linkedDataverse = null; + List alreadyLinkeddv_ids; + + if ((dvo instanceof Dataset)) { + linkedDataset = (Dataset) dvo; + alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + linkedDataset.getId()).getResultList(); + } else { + linkedDataverse = (Dataverse) dvo; + alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM dataverselinkingdataverse WHERE dataverse_id = " + linkedDataverse.getId()).getResultList(); } - List alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + dataset.getId()).getResultList(); List remove = new ArrayList<>(); if (alreadyLinkeddv_ids != null && !alreadyLinkeddv_ids.isEmpty()) { @@ -523,10 +534,15 @@ public List filterDataversesForLinking(String query, DataverseRequest remove.add(removeIt); }); } - + + if (dvo instanceof Dataverse dataverse) { + remove.add(dataverse); + } + for (Dataverse res : results) { if (!remove.contains(res)) { - if (this.permissionService.requestOn(req, res).has(Permission.LinkDataset)) { + if ((linkedDataset != null && this.permissionService.requestOn(req, res).has(Permission.LinkDataset)) + || (linkedDataverse != null && this.permissionService.requestOn(req, res).has(Permission.LinkDataverse))) { dataverseList.add(res); } } @@ -534,6 +550,54 @@ public List filterDataversesForLinking(String query, DataverseRequest return dataverseList; } + + public List removeUnlinkableDataverses(List allWithPerms, DvObject dvo) { + List dataverseList = new ArrayList<>(); + Dataset linkedDataset = null; + Dataverse linkedDataverse = null; + List alreadyLinkeddv_ids; + + if ((dvo instanceof Dataset)) { + linkedDataset = (Dataset) dvo; + alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + linkedDataset.getId()).getResultList(); + } else { + linkedDataverse = (Dataverse) dvo; + alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM dataverselinkingdataverse WHERE dataverse_id = " + linkedDataverse.getId()).getResultList(); + } + + List remove = new ArrayList<>(); + + if (alreadyLinkeddv_ids != null && !alreadyLinkeddv_ids.isEmpty()) { + alreadyLinkeddv_ids.stream().map((testDVId) -> this.find(testDVId)).forEachOrdered((removeIt) -> { + remove.add(removeIt); + }); + } + + + if (dvo instanceof Dataverse dataverse) { + remove.add(dataverse); + } + + DvObject testDVO = dvo; + //Remove DVO's parent up to Root + while (testDVO != null) { + if (testDVO.getOwner() == null) { + break; // we are at the root; which by definition is metadata block root, regardless of the value + } + remove.add((Dataverse) testDVO.getOwner()); + testDVO = testDVO.getOwner(); + } + + for (Dataverse res : allWithPerms) { + if (!remove.contains(res)) { + dataverseList.add(res); + } + } + + return dataverseList; + } + + public List filterDataversesForUnLinking(String query, DataverseRequest req, Dataset dataset) { List alreadyLinkeddv_ids = em.createNativeQuery("SELECT linkingdataverse_id FROM datasetlinkingdataverse WHERE dataset_id = " + dataset.getId()).getResultList(); List dataverseList = new ArrayList<>(); diff --git a/src/main/java/edu/harvard/iq/dataverse/FilePage.java b/src/main/java/edu/harvard/iq/dataverse/FilePage.java index 50ebbc33634..dd1bd56d5bd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FilePage.java +++ b/src/main/java/edu/harvard/iq/dataverse/FilePage.java @@ -420,6 +420,11 @@ public boolean isGuestbookPopupRequiredAtDownload(){ DatasetVersion workingVersion = fileMetadata.getDatasetVersion(); return FileUtil.isGuestbookPopupRequired(workingVersion) && !workingVersion.getDataset().getEffectiveGuestbookEntryAtRequest(); } + + public boolean isGuestbookPopupRequired(){ + DatasetVersion workingVersion = fileMetadata.getDatasetVersion(); + return FileUtil.isGuestbookPopupRequired(workingVersion); + } public void setFileMetadata(FileMetadata fileMetadata) { this.fileMetadata = fileMetadata; diff --git a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java index e00e303356e..d492991bb62 100644 --- a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java @@ -96,7 +96,7 @@ public class PermissionServiceBean { DatasetVersionFilesServiceBean datasetVersionFilesServiceBean; private static final String LIST_ALL_DATAVERSES_SUPERUSER_HAS_PERMISSION = """ - SELECT id, name, alias FROM DATAVERSE + SELECT id, name, alias FROM DATAVERSE dv """; private static final String LIST_ALL_DATAVERSES_USER_HAS_PERMISSION = """ @@ -105,7 +105,7 @@ WITH grouplist AS ( WHERE explicitgroup_authenticateduser.containedauthenticatedusers_id = @USERID ) - SELECT * FROM DATAVERSE WHERE id IN ( + SELECT * FROM DATAVERSE dv WHERE id IN ( SELECT definitionpoint_id FROM roleassignment WHERE roleassignment.assigneeidentifier IN ( @@ -161,8 +161,24 @@ AND EXISTS (SELECT id FROM dataverserole WHERE dataverserole.id = roleassignment AND @IPRANGESQL ) ) - ) + ) """; + + private static final String AND = """ + and + """; + + private static final String WHERE = """ + where + """; + + private static final String SEARCH_PARAMS = """ + ((LOWER(dv.name) LIKE ? and ((SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?) + or (SUBSTRING(LOWER(dv.name),0,(LENGTH(dv.name)-9)) LIKE ?))) + or (LOWER(dv.name) NOT LIKE ? and ((LOWER(dv.name) LIKE ?) + or (LOWER(dv.name) LIKE ?)))) + """; + /** * A request-level permission query (e.g includes IP ras). */ @@ -921,12 +937,21 @@ private boolean hasUnrestrictedReleasedFiles(DatasetVersion targetDatasetVersion Long result = em.createQuery(criteriaQuery).getSingleResult(); return result > 0; } - + public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, Permission permission) { - return findPermittedCollections(request, user, 1 << permission.ordinal()); + return findPermittedCollections(request, user, 1 << permission.ordinal(), ""); } + public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, Permission permission, String searchTerm) { + return findPermittedCollections(request, user, 1 << permission.ordinal(), searchTerm); + } + public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, int permissionBit) { + return findPermittedCollections(request, user, permissionBit, ""); + } + + + public List findPermittedCollections(DataverseRequest request, AuthenticatedUser user, int permissionBit, String searchTerm) { if (user != null) { // IP Group - Only check IP if a User is calling for themself String ipRangeSQL = "FALSE"; @@ -958,13 +983,64 @@ public List findPermittedCollections(DataverseRequest request, Authen String sqlCode; if (user.isSuperuser()) { sqlCode = LIST_ALL_DATAVERSES_SUPERUSER_HAS_PERMISSION; + + if (searchTerm == null || searchTerm.isEmpty()) { + return em.createNativeQuery(sqlCode, Dataverse.class).getResultList(); + } else { + sqlCode = LIST_ALL_DATAVERSES_SUPERUSER_HAS_PERMISSION.concat(WHERE).concat(SEARCH_PARAMS); + + String pattern = searchTerm.toLowerCase(); + String pattern1 = pattern + "%"; + String pattern2 = "% " + pattern + "%"; + + // Adjust the queries for very short, 1 + if (pattern.length() == 1) { + pattern1 = pattern; + pattern2 = pattern + " %"; + } + Query query = em.createNativeQuery(sqlCode, Dataverse.class); + query.setParameter(1, "%dataverse"); + query.setParameter(2, pattern1); + query.setParameter(3, pattern2); + query.setParameter(4, "%dataverse"); + query.setParameter(5, pattern1); + query.setParameter(6, pattern2); + return query.getResultList(); + + } } else { - sqlCode = LIST_ALL_DATAVERSES_USER_HAS_PERMISSION - .replace("@USERID", String.valueOf(user.getId())) - .replace("@PERMISSIONBIT", String.valueOf(permissionBit)) - .replace("@IPRANGESQL", ipRangeSQL); + if (searchTerm == null || searchTerm.isEmpty()) { + sqlCode = LIST_ALL_DATAVERSES_USER_HAS_PERMISSION + .replace("@USERID", String.valueOf(user.getId())) + .replace("@PERMISSIONBIT", String.valueOf(permissionBit)) + .replace("@IPRANGESQL", ipRangeSQL); + return em.createNativeQuery(sqlCode, Dataverse.class).getResultList(); + } else { + String pattern = searchTerm.toLowerCase(); + String pattern1 = pattern + "%"; + String pattern2 = "% " + pattern + "%"; + + // Adjust the queries for very short, 1 + if (pattern.length() == 1) { + pattern1 = pattern; + pattern2 = pattern + " %"; + } + + sqlCode = LIST_ALL_DATAVERSES_USER_HAS_PERMISSION.concat(AND).concat(SEARCH_PARAMS) + .replace("@USERID", String.valueOf(user.getId())) + .replace("@PERMISSIONBIT", String.valueOf(permissionBit)) + .replace("@IPRANGESQL", ipRangeSQL); + + Query query = em.createNativeQuery(sqlCode, Dataverse.class); + query.setParameter(1, "%dataverse"); + query.setParameter(2, pattern1); + query.setParameter(3, pattern2); + query.setParameter(4, "%dataverse"); + query.setParameter(5, pattern1); + query.setParameter(6, pattern2); + return query.getResultList(); + } } - return em.createNativeQuery(sqlCode, Dataverse.class).getResultList(); } return null; } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java b/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java index 46e8263da15..0ce84844289 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java @@ -615,11 +615,12 @@ protected DvObject findDvo(@NotNull final String id) throws WrappedResponse { * * @param dvIdtf * @param type + * @param testForReleased * @return DvObject if type matches or throw exception * @throws WrappedResponse */ @NotNull - protected DvObject findDvoByIdAndFeaturedItemTypeOrDie(@NotNull final String dvIdtf, String type) throws WrappedResponse { + protected DvObject findDvoByIdAndTypeOrDie(@NotNull final String dvIdtf, String type, boolean testForReleased) throws WrappedResponse { try { DataverseFeaturedItem.TYPES dvType = DataverseFeaturedItem.getDvType(type); DvObject dvObject = null; @@ -652,7 +653,12 @@ protected DvObject findDvoByIdAndFeaturedItemTypeOrDie(@NotNull final String dvI } } } - DataverseFeaturedItem.validateTypeAndDvObject(dvIdtf, dvObject, dvType); + if (testForReleased){ + DataverseFeaturedItem.validateTypeAndDvObject(dvIdtf, dvObject, dvType); + } + if (dvObject == null) { + throw new WrappedResponse(notFound(BundleUtil.getStringFromBundle("find.dvo.error.dvObjectNotFound", Collections.singletonList(dvIdtf)))); + } return dvObject; } catch (IllegalArgumentException e) { throw new WrappedResponse(error(Response.Status.BAD_REQUEST, e.getMessage())); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java index d55f582ecae..12989a1b70c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java @@ -20,6 +20,7 @@ import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.util.StringUtil; import edu.harvard.iq.dataverse.util.cache.CacheFactoryBean; +import edu.harvard.iq.dataverse.util.json.JsonPrinter; import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; import edu.harvard.iq.dataverse.validation.EMailValidator; import edu.harvard.iq.dataverse.EjbDataverseEngine; @@ -2180,7 +2181,8 @@ public Response addRoleAssignementsToChildren(@Context ContainerRequestContext c @GET @AuthRequired @Path("/dataverse/{alias}/storageDriver") - public Response getStorageDriver(@Context ContainerRequestContext crc, @PathParam("alias") String alias) throws WrappedResponse { + public Response getStorageDriver(@Context ContainerRequestContext crc, @PathParam("alias") String alias, + @QueryParam("getEffective") Boolean getEffective) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); @@ -2193,8 +2195,12 @@ public Response getStorageDriver(@Context ContainerRequestContext crc, @PathPara } catch (WrappedResponse wr) { return wr.getResponse(); } - //Note that this returns what's set directly on this dataverse. If null/DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER, the user would have to recurse the chain of parents to find the effective storageDriver - return ok(dataverse.getStorageDriverId()); + + if (getEffective != null && getEffective) { + return ok(JsonPrinter.jsonStorageDriver(dataverse.getEffectiveStorageDriverId(), null)); + } else { + return ok(JsonPrinter.jsonStorageDriver(dataverse.getStorageDriverId(), null)); + } } @PUT diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index 10abdb63d82..aa39d1b6c94 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -2608,7 +2608,7 @@ public Response getCurationStatus(@Context ContainerRequestContext crc, canSeeStatus = permissionSvc.requestOn(createDataverseRequest(user), ds).has(Permission.PublishDataset); } - if (dsv.isDraft() && (canSeeStatus)) { + if (canSeeStatus) { List statuses = includeHistory ? dsv.getCurationStatuses() : Collections.singletonList(dsv.getCurrentCurationStatus()); if (includeHistory) { JsonArrayBuilder arrayBuilder = Json.createArrayBuilder(); @@ -3689,7 +3689,7 @@ public Response getFileStore(@Context ContainerRequestContext crc, @PathParam("i return error(Response.Status.NOT_FOUND, "No such dataset"); } - return response(req -> ok(dataset.getEffectiveStorageDriverId()), getRequestUser(crc)); + return ok(JsonPrinter.jsonStorageDriver(dataset.getEffectiveStorageDriverId(), dataset)); } @PUT diff --git a/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java b/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java index 30c3146fbfb..7fbdd79e3c3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/DataverseFeaturedItems.java @@ -63,7 +63,7 @@ public Response updateFeaturedItem(@Context ContainerRequestContext crc, if (dataverseFeaturedItem == null) { throw new WrappedResponse(error(Response.Status.NOT_FOUND, MessageFormat.format(BundleUtil.getStringFromBundle("dataverseFeaturedItems.errors.notFound"), id))); } - DvObject dvObject = (dvObjectIdtf != null) ? findDvoByIdAndFeaturedItemTypeOrDie(dvObjectIdtf, type) : null; + DvObject dvObject = (dvObjectIdtf != null) ? findDvoByIdAndTypeOrDie(dvObjectIdtf, type, true) : null; UpdatedDataverseFeaturedItemDTO updatedDataverseFeaturedItemDTO = UpdatedDataverseFeaturedItemDTO.fromFormData(content, displayOrder, keepFile, imageFileInputStream, contentDispositionHeader, type, dvObject); return ok(json(execCommand(new UpdateDataverseFeaturedItemCommand(createDataverseRequest(getRequestUser(crc)), dataverseFeaturedItem, updatedDataverseFeaturedItemDTO)))); } catch (WrappedResponse e) { diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java index ae82ff46522..7322a8c9341 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java @@ -15,6 +15,7 @@ import edu.harvard.iq.dataverse.authorization.groups.impl.explicit.ExplicitGroup; import edu.harvard.iq.dataverse.authorization.groups.impl.explicit.ExplicitGroupProvider; import edu.harvard.iq.dataverse.authorization.groups.impl.explicit.ExplicitGroupServiceBean; +import edu.harvard.iq.dataverse.authorization.groups.impl.ipaddress.ip.IpAddress; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.dataset.DatasetType; @@ -119,7 +120,7 @@ public class Dataverses extends AbstractApiBean { @EJB DataverseFeaturedItemServiceBean dataverseFeaturedItemServiceBean; - + @POST @AuthRequired public Response addRoot(@Context ContainerRequestContext crc, String body) { @@ -714,7 +715,7 @@ public Response getDataverse(@Context ContainerRequestContext crc, @PathParam("i return response(req -> { Dataverse dataverse = execCommand(new GetDataverseCommand(req, findDataverseOrDie(idtf))); boolean hideEmail = settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false); - return ok(json(dataverse, hideEmail, returnOwners, returnChildCount ? dataverseService.getChildCount(dataverse) : null)); + return ok(json(dataverse, hideEmail, returnOwners, false, returnChildCount ? dataverseService.getChildCount(dataverse) : null)); }, getRequestUser(crc)); } @@ -1771,6 +1772,36 @@ public Response linkDataverse(@Context ContainerRequestContext crc, @PathParam(" return ex.getResponse(); } } + + @GET + @AuthRequired + @Produces(MediaType.APPLICATION_JSON) + @Path("{identifier}/{type}/linkingDataverses") + public Response getLinkingDataverseList(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @QueryParam("searchTerm") String searchTerm, @QueryParam("alreadyLinking") boolean alreadyLinking, @PathParam("type") String type) { + + try { + + DvObject dvObject = findDvoByIdAndTypeOrDie(dvIdtf, type, false); + List dataversesForLinking = execCommand(new GetLinkingDataverseListCommand( + createDataverseRequest(getRequestUser(crc)), + dvObject, + searchTerm, + alreadyLinking + )); + + JsonArrayBuilder dvBuilder = Json.createArrayBuilder(); + if (dataversesForLinking != null && !dataversesForLinking.isEmpty()) { + for (Dataverse dv : dataversesForLinking) { + dvBuilder.add(json(dv, true)); + } + } + return ok(dvBuilder); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + @GET @AuthRequired @@ -1811,7 +1842,7 @@ public Response createFeaturedItem(@Context ContainerRequestContext crc, try { dataverse = findDataverseOrDie(dvIdtf); if (dvObjectIdtf != null) { - dvObject = findDvoByIdAndFeaturedItemTypeOrDie(dvObjectIdtf, type); + dvObject = findDvoByIdAndTypeOrDie(dvObjectIdtf, type, true); } } catch (WrappedResponse wr) { return wr.getResponse(); @@ -1899,7 +1930,7 @@ public Response updateFeaturedItems( // ignore dvObject if the id is missing or an empty string DvObject dvObject = dvObjectIdtf.get(i) != null && !dvObjectIdtf.get(i).isEmpty() - ? findDvoByIdAndFeaturedItemTypeOrDie(dvObjectIdtf.get(i), types.get(i)) : null; + ? findDvoByIdAndTypeOrDie(dvObjectIdtf.get(i), types.get(i), true) : null; if (ids.get(i) == 0) { newItems.add(NewDataverseFeaturedItemDTO.fromFormData( contents.get(i), displayOrders.get(i), fileInputStream, contentDisposition, types.get(i), dvObject)); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java b/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java index 562fd7fcb81..ca4f55da822 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/MakeDataCountApi.java @@ -25,9 +25,15 @@ import java.net.URL; import java.util.Iterator; import java.util.List; +import java.util.concurrent.Future; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.atomic.AtomicLong; import java.util.logging.Level; import java.util.logging.Logger; + +import jakarta.annotation.Resource; import jakarta.ejb.EJB; +import jakarta.enterprise.concurrent.ManagedExecutorService; import jakarta.json.Json; import jakarta.json.JsonArray; import jakarta.json.JsonArrayBuilder; @@ -62,6 +68,13 @@ public class MakeDataCountApi extends AbstractApiBean { @EJB SystemConfig systemConfig; + // Inject the managed executor service provided by the container + @Resource(name = "concurrent/CitationUpdateExecutor") + private ManagedExecutorService executorService; + + // Track the last execution time to implement rate limiting during Citation updates + private static final AtomicLong lastExecutionTime = new AtomicLong(0); + /** * TODO: For each dataset, send the following: * @@ -141,34 +154,109 @@ public Response addUsageMetricsFromSushiReportAll(@QueryParam("reportOnDisk") St @POST @Path("{id}/updateCitationsForDataset") - public Response updateCitationsForDataset(@PathParam("id") String id) throws IOException { + public Response updateCitationsForDataset(@PathParam("id") String id) { try { - Dataset dataset = findDatasetOrDie(id); - GlobalId pid = dataset.getGlobalId(); - PidProvider pidProvider = PidUtil.getPidProvider(pid.getProviderId()); + // First validate that the dataset exists and has a valid DOI + final Dataset dataset = findDatasetOrDie(id); + final GlobalId pid = dataset.getGlobalId(); + final PidProvider pidProvider = PidUtil.getPidProvider(pid.getProviderId()); + // Only supported for DOIs and for DataCite DOI providers - if(!DataCiteDOIProvider.TYPE.equals(pidProvider.getProviderType())) { + if (!DataCiteDOIProvider.TYPE.equals(pidProvider.getProviderType())) { return error(Status.BAD_REQUEST, "Only DataCite DOI providers are supported"); } - String persistentId = pid.toString(); - // DataCite wants "doi=", not "doi:". - String authorityPlusIdentifier = persistentId.replaceFirst("doi:", ""); - // Request max page size and then loop to handle multiple pages - URL url = null; + // Submit the task to the managed executor service + Future future; try { - url = new URI(JvmSettings.DATACITE_REST_API_URL.lookup(pidProvider.getId()) + - "/events?doi=" + - authorityPlusIdentifier + - "&source=crossref&page[size]=1000&page[cursor]=1").toURL(); - } catch (URISyntaxException e) { - //Nominally this means a config error/ bad DATACITE_REST_API_URL for this provider - logger.warning("Unable to create URL for " + persistentId + ", pidProvider " + pidProvider.getId()); - return error(Status.INTERNAL_SERVER_ERROR, "Unable to create DataCite URL to retrieve citations."); + future = executorService.submit(() -> { + try { + // Apply rate limiting if enabled + applyRateLimit(); + + // Process the citation update + boolean success = processCitationUpdate(dataset, pid, pidProvider); + + // Update the last execution time after processing + lastExecutionTime.set(System.currentTimeMillis()); + + if (success) { + logger.fine("Successfully processed citation update for dataset " + id); + } else { + logger.warning("Failed to process citation update for dataset " + id); + } + } catch (Exception e) { + logger.log(Level.SEVERE, "Error processing citation update for dataset " + id, e); + } + }); + + JsonObjectBuilder output = Json.createObjectBuilder(); + output.add("status", "queued"); + output.add("message", "Citation update for dataset " + id + " has been queued for processing"); + return ok(output); + } catch (RejectedExecutionException ree) { + logger.warning("Citation update for dataset " + id + " was rejected: Queue is full"); + return error(Status.SERVICE_UNAVAILABLE, + "Citation update service is currently at capacity. Please try again later."); } - logger.fine("Retrieving Citations from " + url.toString()); - boolean nextPage = true; - JsonArrayBuilder dataBuilder = Json.createArrayBuilder(); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + /** + * Apply rate limiting by waiting if necessary + */ + private void applyRateLimit() { + // Check if rate limiting is enabled + long minDelay = JvmSettings.API_MDC_UPDATE_MIN_DELAY_MS.lookupOptional(Long.class).orElse(0l); + if(minDelay ==0) { + return; + } + // Calculate how long to wait + long lastExecution = lastExecutionTime.get(); + long currentTime = System.currentTimeMillis(); + long elapsedTime = currentTime - lastExecution; + + // If not enough time has passed since the last execution, wait + if (lastExecution > 0 && elapsedTime < minDelay) { + long waitTime = minDelay - elapsedTime; + logger.fine("Rate limiting: waiting " + waitTime + " ms before processing next citation update"); + try { + Thread.sleep(waitTime); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + logger.warning("Rate limiting sleep interrupted: " + e.getMessage()); + } + } + } + + /** + * Process the citation update for a dataset + * This method contains the logic that was previously in updateCitationsForDataset + * @return true if processing was successful, false otherwise + */ + private boolean processCitationUpdate(Dataset dataset, GlobalId pid, PidProvider pidProvider) { + String persistentId = pid.asRawIdentifier(); + + // Request max page size and then loop to handle multiple pages + URL url = null; + try { + url = new URI(JvmSettings.DATACITE_REST_API_URL.lookup(pidProvider.getId()) + + "/events?doi=" + + persistentId + + "&source=crossref&page[size]=1000&page[cursor]=1").toURL(); + } catch (URISyntaxException | MalformedURLException e) { + //Nominally this means a config error/ bad DATACITE_REST_API_URL for this provider + logger.warning("Unable to create URL for " + persistentId + ", pidProvider " + pidProvider.getId()); + return false; + } + + logger.fine("Retrieving Citations from " + url.toString()); + boolean nextPage = true; + JsonArrayBuilder dataBuilder = Json.createArrayBuilder(); + + try { do { HttpURLConnection connection = (HttpURLConnection) url.openConnection(); connection.setRequestMethod("GET"); @@ -176,34 +264,56 @@ public Response updateCitationsForDataset(@PathParam("id") String id) throws IOE if (status != 200) { logger.warning("Failed to get citations from " + url.toString()); connection.disconnect(); - return error(Status.fromStatusCode(status), "Failed to get citations from " + url.toString()); + return false; } + JsonObject report; try (InputStream inStream = connection.getInputStream()) { report = JsonUtil.getJsonObject(inStream); } finally { connection.disconnect(); } + JsonObject links = report.getJsonObject("links"); JsonArray data = report.getJsonArray("data"); Iterator iter = data.iterator(); while (iter.hasNext()) { - dataBuilder.add(iter.next()); + JsonValue citationValue = iter.next(); + JsonObject citation = (JsonObject) citationValue; + + // Filter out relations we don't use (e.g. hasPart) to lower memory req. with many files + if (citation.containsKey("attributes")) { + JsonObject attributes = citation.getJsonObject("attributes"); + if (attributes.containsKey("relation-type-id")) { + String relationshipType = attributes.getString("relation-type-id"); + + // Only add citations with relationship types we care about + if (DatasetExternalCitationsServiceBean.inboundRelationships.contains(relationshipType) || + DatasetExternalCitationsServiceBean.outboundRelationships.contains(relationshipType)) { + dataBuilder.add(citationValue); + } + } + } } + if (links.containsKey("next")) { try { url = new URI(links.getString("next")).toURL(); + applyRateLimit(); } catch (URISyntaxException e) { logger.warning("Unable to create URL from DataCite response: " + links.getString("next")); - return error(Status.INTERNAL_SERVER_ERROR, "Unable to retrieve all results from DataCite"); + return false; } } else { nextPage = false; } + logger.fine("body of citation response: " + report.toString()); } while (nextPage == true); + JsonArray allData = dataBuilder.build(); List datasetExternalCitations = datasetExternalCitationsService.parseCitations(allData); + /* * ToDo: If this is the only source of citations, we should remove all the existing ones for the dataset and repopulate them. * As is, this call doesn't remove old citations if there are now none (legacy issue if we decide to stop counting certain types of citation @@ -216,14 +326,16 @@ public Response updateCitationsForDataset(@PathParam("id") String id) throws IOE datasetExternalCitationsService.save(dm); } } - - JsonObjectBuilder output = Json.createObjectBuilder(); - output.add("citationCount", datasetExternalCitations.size()); - return ok(output); - } catch (WrappedResponse wr) { - return wr.getResponse(); + + logger.fine("Citation update completed for dataset " + dataset.getId() + + " with " + datasetExternalCitations.size() + " citations"); + return true; + } catch (IOException e) { + logger.log(Level.WARNING, "Error processing citation update for dataset " + dataset.getId(), e); + return false; } } + @GET @Path("{yearMonth}/processingState") public Response getProcessingState(@PathParam("yearMonth") String yearMonth) { diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java index 22a3d825cfb..0a0d154a64e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/shib/ShibServiceBean.java @@ -52,7 +52,7 @@ public class ShibServiceBean { private static final String INCOMMON_MDQ_API_BASE = "https://mdq.incommon.org"; private static final String INCOMMON_MDQ_API_ENTITIES_URL = INCOMMON_MDQ_API_BASE + "/entities/"; private static final String INCOMMON_WAYFINDER_URL = "https://wayfinder.incommon.org"; - + private static final String INCOMMON_ORGANIZATION_ATTRIBUTE = "OrganizationDisplayName"; /** * "Production" means "don't mess with the HTTP request". */ @@ -233,7 +233,7 @@ public String getAffiliationViaMDQ(String shibIdp) { if (event == XMLStreamConstants.START_ELEMENT) { String currentElement = xmlr.getLocalName(); - if ("".equals(currentElement)) { + if (INCOMMON_ORGANIZATION_ATTRIBUTE.equals(currentElement)) { int eventType = xmlr.next(); if (eventType == XMLStreamConstants.CHARACTERS) { String affiliation = xmlr.getText(); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java index 8227572da3b..e4fd5373c7d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/AbstractWriteDataverseCommand.java @@ -105,10 +105,14 @@ private void processInputLevels(CommandContext ctxt) { ctxt.fieldTypeInputLevels().deleteDataverseFieldTypeInputLevelFor(dataverse); } else { dataverse.addInputLevelsMetadataBlocksIfNotPresent(inputLevels); - ctxt.fieldTypeInputLevels().deleteDataverseFieldTypeInputLevelFor(dataverse); + //if levels not empty either create or update (handled by save - update when id not null create if null) inputLevels.forEach(inputLevel -> { + DataverseFieldTypeInputLevel ftil = ctxt.fieldTypeInputLevels().findByDataverseIdDatasetFieldTypeId(dataverse.getId(), inputLevel.getDatasetFieldType().getId()); + if(ftil != null){ + inputLevel.setId(ftil.getId()); + } inputLevel.setDataverse(dataverse); - ctxt.fieldTypeInputLevels().create(inputLevel); + ctxt.fieldTypeInputLevels().save(inputLevel); }); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java index 3629432b7e4..9ebdd28e009 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java @@ -15,6 +15,7 @@ import edu.harvard.iq.dataverse.DatasetField; import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.TermsOfUseAndAccess; +import edu.harvard.iq.dataverse.CurationStatus; import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.RoleAssignment; @@ -27,6 +28,8 @@ import java.util.logging.Level; import java.util.logging.Logger; +import org.apache.commons.lang3.StringUtils; + /** * * @author qqmyers @@ -73,7 +76,7 @@ public Dataset execute(CommandContext ctxt) throws CommandException { newTerms.setDatasetVersion(updateVersion); updateVersion.setTermsOfUseAndAccess(newTerms); - //Creation Note + //Version Note updateVersion.setVersionNote(newVersion.getVersionNote()); // Clear unnecessary terms relationships .... @@ -96,6 +99,24 @@ public Dataset execute(CommandContext ctxt) throws CommandException { updateVersion.getWorkflowComments().addAll(newComments); } + // Transfer curation status entries from draft to published version + List draftCurationStatuses = newVersion.getCurationStatuses(); + if (draftCurationStatuses != null && !draftCurationStatuses.isEmpty()) { + for (CurationStatus cs : draftCurationStatuses) { + // Update the dataset version reference + //This call sets the version in the curationstatus object as well + updateVersion.addCurationStatus(cs); + } + // Clear the list from the draft version + newVersion.getCurationStatuses().clear(); + } + + // Add a new empty curation status to clear the status in the published version (as done in the FinalizeDatasetPublicationCommand) + CurationStatus status = updateVersion.getCurrentCurationStatus(); + if (status != null && StringUtils.isNotBlank(status.getLabel())) { + updateVersion.addCurationStatus(new CurationStatus(null, updateVersion, getRequest().getAuthenticatedUser())); + } + // we have to merge to update the database but not flush because // we don't want to create two draft versions! Dataset tempDataset = getDataset(); @@ -201,7 +222,7 @@ public Dataset execute(CommandContext ctxt) throws CommandException { // This can be corrected by running the update PID API later, but who will look in the log? // With the change to not use the DeleteDatasetVersionCommand above and other // fixes, this error may now cleanly restore the initial state - // with the draft and last published versions unchanged, but this has not yet bee tested. + // with the draft and last published versions unchanged, but this has not yet been tested. // (Alternately this could move to onSuccess if we intend it to stay non-fatal.) logger.log(Level.WARNING, "Curate Published DatasetVersion: exception while updating PID metadata:{0}", ex.getMessage()); } diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java new file mode 100644 index 00000000000..466871b88e4 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommand.java @@ -0,0 +1,113 @@ + +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DvObject; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.engine.command.AbstractCommand; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.RequiredPermissions; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; +import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; +import edu.harvard.iq.dataverse.util.BundleUtil; +import java.util.ArrayList; +import java.util.EnumSet; +import java.util.List; + +/** + * + * @author stephenkraffmiller + */ +@RequiredPermissions({}) +public class GetLinkingDataverseListCommand extends AbstractCommand> { + + private final String searchTerm; + private final DvObject dvObject; + private final boolean alreadyLinked; + + public GetLinkingDataverseListCommand(DataverseRequest aRequest, DvObject dvObject, String searchTerm) { + super(aRequest, dvObject); + this.searchTerm = searchTerm; + this.dvObject = dvObject; + this.alreadyLinked = false; + } + + public GetLinkingDataverseListCommand(DataverseRequest aRequest, DvObject dvObject, boolean alreadyLinked) { + super(aRequest, dvObject); + this.searchTerm = ""; + this.dvObject = dvObject; + this.alreadyLinked = alreadyLinked; + } + + public GetLinkingDataverseListCommand(DataverseRequest aRequest, DvObject dvObject, String searchTerm, boolean alreadyLinked) { + super(aRequest, dvObject); + this.searchTerm = searchTerm; + this.dvObject = dvObject; + this.alreadyLinked = alreadyLinked; + } + + + + @Override + public List execute(CommandContext ctxt) throws CommandException { + + User requestUser = (User) getRequest().getUser(); + AuthenticatedUser authUser; + if (!requestUser.isAuthenticated()) { + throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataverse.link.user"), this); + } else { + authUser = (AuthenticatedUser) requestUser; + } + List dataversesForLinking; + String searchParam; + if (searchTerm != null) { + searchParam = searchTerm; + } else { + searchParam = ""; + } + + //Find Permitted Collections now takes a Search Term to filter down collections the user may link + Permission permToCheck; + if (dvObject instanceof Dataset) { + permToCheck = Permission.LinkDataset; + } else { + permToCheck = Permission.LinkDataverse; + } + //depending on the already linked boolean the command will return a list of Dataverses available for linking + // or a list of dataverses to which the object has already been linked - for the unlink function + if (!alreadyLinked) { + dataversesForLinking = ctxt.permissions().findPermittedCollections(getRequest(), authUser, permToCheck, searchParam); + //Don't bother with checking for already linked if there are none to be tested. + if (dataversesForLinking == null || dataversesForLinking.isEmpty()) { + return dataversesForLinking; + } + return ctxt.dataverses().removeUnlinkableDataverses(dataversesForLinking, dvObject); + } else { + List dataversesAlreadyLinked; + List dataversesAlreadyLinkedCanUnlink = new ArrayList<>(); + //new ArrayList<>(); + if (dvObject instanceof Dataset) { + dataversesAlreadyLinked = ctxt.dsLinking().findLinkingDataverses(dvObject.getId(), searchParam); + for (Dataverse dv : dataversesAlreadyLinked) { + if (ctxt.permissions().hasPermissionsFor(getRequest(), dv, EnumSet.of(Permission.LinkDataset))) { + dataversesAlreadyLinkedCanUnlink.add(dv); + } + return dataversesAlreadyLinkedCanUnlink; + } + } else { + dataversesAlreadyLinked = ctxt.dvLinking().findLinkingDataverses(dvObject.getId(), searchParam); + for (Dataverse dv : dataversesAlreadyLinked) { + if (ctxt.permissions().hasPermissionsFor(getRequest(), dv, EnumSet.of(Permission.LinkDataverse))) { + dataversesAlreadyLinkedCanUnlink.add(dv); + } + } + return dataversesAlreadyLinkedCanUnlink; + } + return null; + } + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java b/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java index b98f88e386f..e7bcf17d44b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java +++ b/src/main/java/edu/harvard/iq/dataverse/export/ExportService.java @@ -312,6 +312,7 @@ public void exportAllFormats(Dataset dataset) throws ExportException { } catch (ServiceConfigurationError serviceError) { throw new ExportException("Service configuration error during export. " + serviceError.getMessage()); } catch (RuntimeException e) { + e.printStackTrace(); logger.log(Level.FINE, e.getMessage(), e); throw new ExportException( "Unknown runtime exception exporting metadata. " + (e.getMessage() == null ? "" : e.getMessage())); diff --git a/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java index fa56432cc3c..fa87926210f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetExternalCitationsServiceBean.java @@ -39,13 +39,13 @@ public class DatasetExternalCitationsServiceBean implements java.io.Serializable DatasetServiceBean datasetService; //Array of relationship types that are considered to be citations - static ArrayList inboundRelationships = new ArrayList( + public static ArrayList inboundRelationships = new ArrayList( Arrays.asList( "cites", "references", "supplements", "is-supplement-to")); - static ArrayList outboundRelationships = new ArrayList( + public static ArrayList outboundRelationships = new ArrayList( Arrays.asList( "is-cited-by", "is-referenced-by", diff --git a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java index baf8302437d..9ebb346baf8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java +++ b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/XmlMetadataTemplate.java @@ -58,6 +58,9 @@ import edu.harvard.iq.dataverse.util.xml.XmlWriterUtil; import jakarta.enterprise.inject.spi.CDI; import jakarta.json.JsonObject; +import jakarta.json.JsonString; +import jakarta.json.JsonValue; +import jakarta.json.JsonValue.ValueType; public class XmlMetadataTemplate { @@ -621,8 +624,12 @@ private void writeEntityElements(XMLStreamWriter xmlw, String elementName, Strin if (externalIdentifier.isValidIdentifier(orgName)) { isROR = true; JsonObject jo = getExternalVocabularyValue(orgName); - if (jo != null) { - orgName = jo.getString("termName"); + // Some ext. cvv configs store a JsonArray of multiple objects/values. In such cases, we'll leave orgName blank + if (jo != null && jo.containsKey("termName")) { + JsonValue termName = jo.get("termName"); + if (termName.getValueType() == ValueType.STRING) { + orgName = ((JsonString) termName).getString(); + } } } diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java index 64679b05beb..aec352a615b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SolrIndexServiceBean.java @@ -28,6 +28,8 @@ import jakarta.ejb.EJB; import jakarta.ejb.Stateless; +import jakarta.ejb.TransactionAttribute; +import jakarta.ejb.TransactionAttributeType; import jakarta.inject.Named; import jakarta.json.Json; import jakarta.json.JsonObjectBuilder; @@ -43,6 +45,9 @@ public class SolrIndexServiceBean { private static final Logger logger = Logger.getLogger(SolrIndexServiceBean.class.getCanonicalName()); + @EJB + private SolrIndexServiceBean self; // Self-injection to allow calling methods in new transactions (from other methods in this bean) + @EJB DvObjectServiceBean dvObjectService; @EJB @@ -317,7 +322,7 @@ private void persistToSolr(Collection docs) throws SolrServer /** * We use the database to determine direct children since there is no - * inheritance + * inheritance. This implementation uses smaller transactions to avoid memory issues. */ public IndexResponse indexPermissionsOnSelfAndChildren(DvObject definitionPoint) { @@ -325,134 +330,150 @@ public IndexResponse indexPermissionsOnSelfAndChildren(DvObject definitionPoint) logger.log(Level.WARNING, "Cannot perform indexPermissionsOnSelfAndChildren with a definitionPoint null"); return null; } - int fileQueryMin= JvmSettings.MIN_FILES_TO_USE_PROXY.lookupOptional(Integer.class).orElse(Integer.MAX_VALUE); - List filesToReindexAsBatch = new ArrayList<>(); - /** - * @todo Re-indexing the definition point itself seems to be necessary - * for revoke but not necessarily grant. - */ - - // We don't create a Solr "primary/content" doc for the root dataverse - // so don't create a Solr "permission" doc either. - final int[] counter = {0}; + int fileQueryMin = JvmSettings.MIN_FILES_TO_USE_PROXY.lookupOptional(Integer.class).orElse(Integer.MAX_VALUE); + final int[] counter = { 0 }; int numObjects = 0; long globalStartTime = System.currentTimeMillis(); - if (definitionPoint.isInstanceofDataverse()) { - Dataverse selfDataverse = (Dataverse) definitionPoint; - if (!selfDataverse.equals(dataverseService.findRootDataverse())) { + + // Handle the definition point itself in its own transaction + if (definitionPoint instanceof Dataverse dataverse) { + // We don't create a Solr "primary/content" doc for the root dataverse + // so don't create a Solr "permission" doc either. + if (!dataverse.equals(dataverseService.findRootDataverse())) { indexPermissionsForOneDvObject(definitionPoint); numObjects++; } - List directChildDatasetsOfDvDefPoint = datasetService.findByOwnerId(selfDataverse.getId()); - for (Dataset dataset : directChildDatasetsOfDvDefPoint) { + + // Process datasets in batches + List datasetIds = datasetService.findIdsByOwnerId(dataverse.getId()); + int batchSize = 10; // Process 10 datasets per transaction + + for (int i = 0; i < datasetIds.size(); i += batchSize) { + int endIndex = Math.min(i + batchSize, datasetIds.size()); + List batchIds = datasetIds.subList(i, endIndex); + + // Process this batch of datasets in a new transaction + self.indexDatasetBatchInNewTransaction(batchIds, counter, fileQueryMin); + numObjects += batchIds.size(); + + logger.fine("Permission reindexing: Processed batch " + (i/batchSize + 1) + " of " + + (int) Math.ceil(datasetIds.size() / (double) batchSize) + + " dataset batches for dataverse " + dataverse.getId()); + } + } else if (definitionPoint instanceof Dataset dataset) { + // For a single dataset, process it in its own transaction + indexPermissionsForOneDvObject(definitionPoint); + numObjects++; + + // Process the dataset's files in a new transaction + self.indexDatasetFilesInNewTransaction(dataset.getId(), counter, fileQueryMin); + } else { + // For other types (like files), just index in a new transaction + indexPermissionsForOneDvObject(definitionPoint); + numObjects++; + } + + logger.fine("Reindexed permissions for " + counter[0] + " files and " + numObjects + + " datasets/collections in " + (System.currentTimeMillis() - globalStartTime) + " ms"); + + return new IndexResponse("Number of dvObject permissions indexed for " + definitionPoint + ": " + numObjects); + } + + @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) + public void indexDatasetBatchInNewTransaction(List datasetIds, final int[] fileCounter, int fileQueryMin) { + for (Long datasetId : datasetIds) { + Dataset dataset = datasetService.find(datasetId); + if (dataset != null) { indexPermissionsForOneDvObject(dataset); - numObjects++; + // Process files for this dataset Map desiredCards = searchPermissionsService.getDesiredCards(dataset); - long startTime = System.currentTimeMillis(); + for (DatasetVersion version : versionsToReIndexPermissionsFor(dataset)) { if (desiredCards.get(version.getVersionState())) { - List cachedPerms = searchPermissionsService.findDatasetVersionPerms(version); - String solrIdEnd = getDatasetOrDataFileSolrEnding(version.getVersionState()); - Long versionId = version.getId(); - for (FileMetadata fmd : version.getFileMetadatas()) { - DataFileProxy fileProxy = new DataFileProxy(fmd); - // Since reindexFilesInBatches() re-indexes a file in all versions needed, we should not send a file already in the released version twice - filesToReindexAsBatch.add(fileProxy); - counter[0]++; - if (counter[0] % 100 == 0) { - reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); - filesToReindexAsBatch.clear(); - } - if (counter[0] % 1000 == 0) { - logger.fine("Progress: " + counter[0] + "files permissions reindexed"); - } - } - - // Re-index any remaining files in the datasetversion (so that verionId, etc. remain constants for all files in the batch) - reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); - logger.info("Progress : dataset " + dataset.getId() + " permissions reindexed in " + (System.currentTimeMillis() - startTime) + " ms"); + processDatasetVersionFiles(version, fileCounter, fileQueryMin); } } } - } else if (definitionPoint.isInstanceofDataset()) { - indexPermissionsForOneDvObject(definitionPoint); - numObjects++; - // index files - Dataset dataset = (Dataset) definitionPoint; + } + } + + @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) + public void indexDatasetFilesInNewTransaction(Long datasetId, final int[] fileCounter, int fileQueryMin) { + Dataset dataset = datasetService.find(datasetId); + if (dataset != null) { Map desiredCards = searchPermissionsService.getDesiredCards(dataset); for (DatasetVersion version : versionsToReIndexPermissionsFor(dataset)) { if (desiredCards.get(version.getVersionState())) { - List cachedPerms = searchPermissionsService.findDatasetVersionPerms(version); - String solrIdEnd = getDatasetOrDataFileSolrEnding(version.getVersionState()); - Long versionId = version.getId(); - if (version.getFileMetadatas().size() > fileQueryMin) { - // For large datasets, use a more efficient SQL query instead of loading all file metadata objects - getDataFileInfoForPermissionIndexing(version.getId()).forEach(fileInfo -> { - filesToReindexAsBatch.add(fileInfo); - counter[0]++; - - if (counter[0] % 100 == 0) { - long startTime = System.currentTimeMillis(); - reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); - filesToReindexAsBatch.clear(); - logger.fine("Progress: 100 file permissions at " + counter[0] + " files reindexed in " + (System.currentTimeMillis() - startTime) + " ms"); - } - }); - } else { - version.getFileMetadatas().stream() - .forEach(fmd -> { - DataFileProxy fileProxy = new DataFileProxy(fmd); - filesToReindexAsBatch.add(fileProxy); - counter[0]++; - if (counter[0] % 100 == 0) { - long startTime = System.currentTimeMillis(); - reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); - filesToReindexAsBatch.clear(); - logger.fine("Progress: 100 file permissions at " + counter[0] + "files reindexed in " + (System.currentTimeMillis() - startTime) + " ms"); - } - }); - } - // Re-index any remaining files in the dataset version (versionId, etc. remain constants for all files in the batch) + processDatasetVersionFiles(version, fileCounter, fileQueryMin); + } + } + } + } + + private void processDatasetVersionFiles(DatasetVersion version, + final int[] fileCounter, int fileQueryMin) { + List cachedPerms = searchPermissionsService.findDatasetVersionPerms(version); + String solrIdEnd = getDatasetOrDataFileSolrEnding(version.getVersionState()); + Long versionId = version.getId(); + List filesToReindexAsBatch = new ArrayList<>(); + + // Process files in batches of 100 + int batchSize = 100; + + if (version.getFileMetadatas().size() > fileQueryMin) { + // For large datasets, use a more efficient SQL query + Stream fileStream = getDataFileInfoForPermissionIndexing(version.getId()); + + // Process files in batches to avoid memory issues + fileStream.forEach(fileInfo -> { + filesToReindexAsBatch.add(fileInfo); + fileCounter[0]++; + + if (filesToReindexAsBatch.size() >= batchSize) { reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); filesToReindexAsBatch.clear(); } + }); + } else { + // For smaller datasets, process files directly + for (FileMetadata fmd : version.getFileMetadatas()) { + DataFileProxy fileProxy = new DataFileProxy(fmd); + filesToReindexAsBatch.add(fileProxy); + fileCounter[0]++; + if (filesToReindexAsBatch.size() >= batchSize) { + reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); + filesToReindexAsBatch.clear(); + } } - } else { - indexPermissionsForOneDvObject(definitionPoint); - numObjects++; } - /** - * @todo Error handling? What to do with response? - * - * @todo Should update timestamps, probably, even thought these are files, see - * https://github.com/IQSS/dataverse/issues/2421 - */ - logger.fine("Reindexed permissions for " + counter[0] + " files and " + numObjects + "datasets/collections in " + (System.currentTimeMillis() - globalStartTime) + " ms"); - return new IndexResponse("Number of dvObject permissions indexed for " + definitionPoint - + ": " + numObjects); + // Process any remaining files + if (!filesToReindexAsBatch.isEmpty()) { + reindexFilesInBatches(filesToReindexAsBatch, cachedPerms, versionId, solrIdEnd); + } } - private String reindexFilesInBatches(List filesToReindexAsBatch, List cachedPerms, Long versionId, String solrIdEnd) { + private void reindexFilesInBatches(List filesToReindexAsBatch, List cachedPerms, Long versionId, String solrIdEnd) { List docs = new ArrayList<>(); try { // Assume all files have the same owner if (filesToReindexAsBatch.isEmpty()) { - return "No files to reindex"; + logger.warning("reindexFilesInBatches called incorrectly with an empty file list"); } - for (DataFileProxy file : filesToReindexAsBatch) { + for (DataFileProxy file : filesToReindexAsBatch) { - DvObjectSolrDoc fileSolrDoc = constructDatafileSolrDoc(file, cachedPerms, versionId, solrIdEnd); - SolrInputDocument solrDoc = SearchUtil.createSolrDoc(fileSolrDoc); - docs.add(solrDoc); - } + DvObjectSolrDoc fileSolrDoc = constructDatafileSolrDoc(file, cachedPerms, versionId, solrIdEnd); + SolrInputDocument solrDoc = SearchUtil.createSolrDoc(fileSolrDoc); + docs.add(solrDoc); + } persistToSolr(docs); - return " " + filesToReindexAsBatch.size() + " files indexed across " + docs.size() + " Solr documents "; + logger.fine("Indexed " + filesToReindexAsBatch.size() + " files across " + docs.size() + " Solr documents"); } catch (SolrServerException | IOException ex) { - return " tried to reindex " + filesToReindexAsBatch.size() + " files indexed across " + docs.size() + " Solr documents but caught exception: " + ex; + logger.log(Level.WARNING, "Failed to reindex " + filesToReindexAsBatch.size() + + " files across " + docs.size() + " Solr documents", ex); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java index 53dff244ae1..87123801a3e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java @@ -93,6 +93,10 @@ public enum JvmSettings { API_BLOCKED_ENDPOINTS(SCOPE_API_BLOCKED, "endpoints"), API_BLOCKED_POLICY(SCOPE_API_BLOCKED, "policy"), API_BLOCKED_KEY(SCOPE_API_BLOCKED, "key"), + // API: MDC Citation updates + SCOPE_API_MDC(SCOPE_API, "mdc"), + API_MDC_UPDATE_MIN_DELAY_MS(SCOPE_API_MDC, "min-delay-ms"), + // SIGNPOSTING SETTINGS SCOPE_SIGNPOSTING(PREFIX, "signposting"), diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index bbc834e0cc4..8b0ea201aa3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -40,6 +40,7 @@ import edu.harvard.iq.dataverse.workflow.Workflow; import edu.harvard.iq.dataverse.workflow.step.WorkflowStepData; +import java.io.IOException; import java.util.*; import jakarta.json.Json; import jakarta.json.JsonArrayBuilder; @@ -290,22 +291,35 @@ public static JsonObjectBuilder json(Workflow wf){ return bld; } + + public static JsonObjectBuilder json(Dataverse dv, boolean minimal) { + if (!minimal){ + return json(dv, false, false, false, null); + } else { + return json(dv, false, false, true, null); + } + } public static JsonObjectBuilder json(Dataverse dv) { - return json(dv, false, false, null); + return json(dv, false, false, false, null); } //TODO: Once we upgrade to Java EE 8 we can remove objects from the builder, and this email removal can be done in a better place. - public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean returnOwners, Long childCount) { + public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean returnOwners, Boolean minimal, Long childCount) { JsonObjectBuilder bld = jsonObjectBuilder() .add("id", dv.getId()) .add("alias", dv.getAlias()) - .add("name", dv.getName()) - .add("affiliation", dv.getAffiliation()); - if(!hideEmail) { + .add("name", dv.getName()); + //minimal refers to only returning the id alias and name for + //used in selecting collections available for linking + if (minimal) { + return bld; + } + bld.add("affiliation", dv.getAffiliation()); + if (!hideEmail) { bld.add("dataverseContacts", JsonPrinter.json(dv.getDataverseContacts())); } - if (returnOwners){ + if (returnOwners) { bld.add("isPartOf", getOwnersFromDvObject(dv)); } bld.add("permissionRoot", dv.isPermissionRoot()) @@ -322,8 +336,8 @@ public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean re if (dv.getDataverseTheme() != null) { bld.add("theme", JsonPrinter.json(dv.getDataverseTheme())); } - if(dv.getStorageDriverId() != null) { - bld.add("storageDriverLabel", DataAccess.getStorageDriverLabelFor(dv.getStorageDriverId())); + if (dv.getStorageDriverId() != null) { + bld.add("storageDriverLabel", DataAccess.getStorageDriverLabelFor(dv.getStorageDriverId())); } if (dv.getFilePIDsEnabled() != null) { bld.add("filePIDsEnabled", dv.getFilePIDsEnabled()); @@ -332,7 +346,7 @@ public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean re bld.add("isReleased", dv.isReleased()); List inputLevels = dv.getDataverseFieldTypeInputLevels(); - if(!inputLevels.isEmpty()) { + if (!inputLevels.isEmpty()) { bld.add("inputLevels", JsonPrinter.jsonDataverseFieldTypeInputLevels(inputLevels)); } @@ -340,7 +354,6 @@ public static JsonObjectBuilder json(Dataverse dv, Boolean hideEmail, Boolean re bld.add("childCount", childCount); } addDatasetFileCountLimit(dv, bld); - return bld; } @@ -1653,6 +1666,23 @@ public static JsonArrayBuilder jsonTemplateInstructions(Map temp return jsonArrayBuilder; } + public static JsonObjectBuilder jsonStorageDriver(String storageDriverId, Dataset dataset) { + JsonObjectBuilder jsonObjectBuilder = new NullSafeJsonBuilder(); + jsonObjectBuilder.add("name", storageDriverId); + jsonObjectBuilder.add("type", DataAccess.getDriverType(storageDriverId)); + jsonObjectBuilder.add("label", DataAccess.getStorageDriverLabelFor(storageDriverId)); + if (dataset != null) { + jsonObjectBuilder.add("directUpload", DataAccess.uploadToDatasetAllowed(dataset, storageDriverId)); + try { + jsonObjectBuilder.add("directDownload", DataAccess.getStorageIO(dataset).downloadRedirectEnabled()); + } catch (IOException ex) { + logger.fine("Failed to get Storage IO for dataset " + ex.getMessage()); + } + } + + return jsonObjectBuilder; + } + public static JsonArrayBuilder json(List notifications, AuthenticatedUser authenticatedUser, boolean inAppNotificationFormat) { JsonArrayBuilder notificationsArray = Json.createArrayBuilder(); diff --git a/src/main/webapp/WEB-INF/glassfish-resources.xml b/src/main/webapp/WEB-INF/glassfish-resources.xml index 3fbbf4c3586..74af3be42ce 100644 --- a/src/main/webapp/WEB-INF/glassfish-resources.xml +++ b/src/main/webapp/WEB-INF/glassfish-resources.xml @@ -11,4 +11,14 @@ + + + + + + + + \ No newline at end of file diff --git a/src/main/webapp/dataset.xhtml b/src/main/webapp/dataset.xhtml index dbc1fb3aeae..7729b6da442 100644 --- a/src/main/webapp/dataset.xhtml +++ b/src/main/webapp/dataset.xhtml @@ -525,7 +525,7 @@ #{bundle['dataset.curationStatusMenu']}
        - +
      • #{bundle['dataset.changestatus']}
          diff --git a/src/main/webapp/file.xhtml b/src/main/webapp/file.xhtml index 8a5c2709e5d..2292ebf4c45 100644 --- a/src/main/webapp/file.xhtml +++ b/src/main/webapp/file.xhtml @@ -374,6 +374,7 @@ + @@ -681,6 +682,7 @@ + diff --git a/src/main/webapp/guestbook-terms-popup-fragment.xhtml b/src/main/webapp/guestbook-terms-popup-fragment.xhtml index 7d157fcb72c..a17145a43ce 100644 --- a/src/main/webapp/guestbook-terms-popup-fragment.xhtml +++ b/src/main/webapp/guestbook-terms-popup-fragment.xhtml @@ -142,7 +142,7 @@ value="#{MarkupChecker:sanitizeBasicHTML(workingVersion.termsOfUseAndAccess.termsOfAccess)}" escape="false" /> - +
          diff --git a/src/main/webapp/resources/css/structure.css b/src/main/webapp/resources/css/structure.css index 4d83ddd489d..39e96390dc7 100644 --- a/src/main/webapp/resources/css/structure.css +++ b/src/main/webapp/resources/css/structure.css @@ -7,6 +7,9 @@ html { body { /* Sticky footer Margin bottom by footer height */ margin-bottom:160px; + /* #11824 */ + word-break: normal; + overflow-wrap: anywhere; } body.widget-view {margin-bottom:44px;} body .ui-widget {font-size: inherit;} diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java index b273502e6ed..16552869a80 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java @@ -3875,6 +3875,8 @@ public void testCurationStatusAPIs() { createDataverseResponse.prettyPrint(); String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse); + UtilIT.publishDataverseViaNativeApi(dataverseAlias, apiToken).then().assertThat().statusCode(OK.getStatusCode()); + Response setCurationLabelSets = UtilIT.setSetting(SettingsServiceBean.Key.AllowedCurationLabels, "{\"StandardProcess\":[\"Author contacted\", \"Privacy Review\", \"Awaiting paper publication\", \"Final Approval\"],\"AlternateProcess\":[\"State 1\",\"State 2\",\"State 3\"]}"); setCurationLabelSets.then().assertThat() .statusCode(OK.getStatusCode()); @@ -3885,7 +3887,7 @@ public void testCurationStatusAPIs() { Response setDataverseCurationLabelSetResponse = UtilIT.setDataverseCurationLabelSet(dataverseAlias, apiToken, "AlternateProcess"); setDataverseCurationLabelSetResponse.then().assertThat().statusCode(FORBIDDEN.getStatusCode()); - Response makeSuperUser = UtilIT.makeSuperUser(username); + Response makeSuperUser = UtilIT.setSuperuserStatus(username, true); assertEquals(200, makeSuperUser.getStatusCode()); //Non-existent option @@ -3944,8 +3946,66 @@ public void testCurationStatusAPIs() { Response setInvalidStatus = UtilIT.setDatasetCurationLabel(datasetId, apiToken, "Invalid Status"); setInvalidStatus.then().assertThat().statusCode(BAD_REQUEST.getStatusCode()); + // Publish the dataset + UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken).then().assertThat() + .statusCode(OK.getStatusCode()); + + // Verify that the current curation label is now empty after publishing + Response getStatusAfterPublish = UtilIT.getDatasetCurationStatus(datasetId, apiToken, false); + getStatusAfterPublish.then().assertThat().statusCode(OK.getStatusCode()); + JsonObject statusAfterPublish = Json.createReader(new StringReader(getStatusAfterPublish.body().asString())).readObject(); + JsonObject dataObject = statusAfterPublish.getJsonObject("data"); + assertFalse(dataObject.containsKey("label"), "Curation label should be empty after publishing"); + + //Cause a new draft version + String jsonLDTerms = "{\"https://dataverse.org/schema/core#fileTermsOfAccess\":{\"https://dataverse.org/schema/core#dataAccessPlace\":\"Somewhere\"}}"; + Response updateTerms = UtilIT.updateDatasetJsonLDMetadata(datasetId, apiToken, jsonLDTerms, true); + updateTerms.then().assertThat() + .statusCode(OK.getStatusCode()); + + // Add a new valid curation label + Response setNewStatus = UtilIT.setDatasetCurationLabel(datasetId, apiToken, "State 2"); + setNewStatus.then().assertThat().statusCode(OK.getStatusCode()); + + // Verify the label was set + Response getStatusAfterSet = UtilIT.getDatasetCurationStatus(datasetId, apiToken, false); + getStatusAfterSet.then().assertThat().statusCode(OK.getStatusCode()); + JsonObject statusAfterSet = Json.createReader(new StringReader(getStatusAfterSet.body().asString())).readObject(); + JsonObject dataInSecondDraft = statusAfterSet.getJsonObject("data"); + assertEquals("State 2", dataInSecondDraft.getString("label"), "Curation label should be set to State 2"); + + // Publish the dataset again using updatecurrent as superuser + Response updateCurrentResponse = UtilIT.publishDatasetViaNativeApi(datasetId, "updatecurrent", apiToken); + updateCurrentResponse.then().assertThat().statusCode(OK.getStatusCode()); + + // Verify that the current curation label is now empty after updatecurrent + Response getStatusAfterUpdateCurrent = UtilIT.getDatasetCurationStatus(datasetId, apiToken, false); + getStatusAfterUpdateCurrent.then().assertThat().statusCode(OK.getStatusCode()); + JsonObject statusAfterUpdateCurrent = Json.createReader(new StringReader(getStatusAfterUpdateCurrent.body().asString())).readObject(); + JsonObject dataAfterUpdateCurrent = statusAfterUpdateCurrent.getJsonObject("data"); + assertFalse(dataAfterUpdateCurrent.containsKey("label"), "Curation label should be empty after updatecurrent"); + + // Verify that the history contains the previously added label + Response getHistoryAfterUpdateCurrent = UtilIT.getDatasetCurationStatus(datasetId, apiToken, true); + getHistoryAfterUpdateCurrent.then().assertThat().statusCode(OK.getStatusCode()); + + // Extract the data array from the response + JsonObject responseObj = Json.createReader(new StringReader(getHistoryAfterUpdateCurrent.body().asString())).readObject(); + JsonArray historyAfterUpdateCurrent = responseObj.getJsonArray("data"); + + // Verify history contains the State 2 label + boolean foundState2 = false; + for (int i = 0; i < historyAfterUpdateCurrent.size(); i++) { + JsonObject entry = historyAfterUpdateCurrent.getJsonObject(i); + if (entry.containsKey("label") && "State 2".equals(entry.getString("label"))) { + foundState2 = true; + break; + } + } + assertTrue(foundState2, "History should contain the State 2 label after updatecurrent"); + // Clean up - Response deleteDatasetResponse = UtilIT.deleteDatasetViaNativeApi(datasetId, apiToken); + Response deleteDatasetResponse = UtilIT.destroyDataset(datasetId, apiToken); assertEquals(200, deleteDatasetResponse.getStatusCode()); Response deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAlias, apiToken); @@ -6058,6 +6118,60 @@ public void testGetGlobusUploadParameters() { GlobusOverlayAccessIOTest.tearDown(); } + @Test + public void testSetGetDatasetStorageDriver() { + Response createUser = UtilIT.createRandomUser(); + createUser.then().assertThat().statusCode(OK.getStatusCode()); + String apiToken = UtilIT.getApiTokenFromResponse(createUser); + String username = UtilIT.getUsernameFromResponse(createUser); + Response makeSuperUser = UtilIT.makeSuperUser(username); + assertEquals(200, makeSuperUser.getStatusCode()); + Response createDataverse = UtilIT.createRandomDataverse(apiToken); + String dataverseAlias = UtilIT.getAliasFromResponse(createDataverse); + Response createDataset = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); + Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + + Response storageDrivers = UtilIT.listStorageDrivers(apiToken); + storageDrivers.prettyPrint(); + JsonObject data = JsonUtil.getJsonObject(storageDrivers.getBody().asString()); + String first = data.getJsonObject("data").keySet().iterator().next(); + String name = data.getJsonObject("data").getString(first); + + Response setDriver = UtilIT.setDatasetStorageDriver(datasetId, first, apiToken); + setDriver.prettyPrint(); + assertEquals(200, setDriver.getStatusCode()); + Response getDriver = UtilIT.getDatasetStorageDriver(datasetId, apiToken); + getDriver.prettyPrint(); + assertEquals(200, getDriver.getStatusCode()); + getDriver.then().assertThat() + .body("data.name", CoreMatchers.equalTo(name)) + .body("data.type", CoreMatchers.notNullValue()) + .body("data.label", CoreMatchers.notNullValue()) + .body("data.directUpload", CoreMatchers.notNullValue()) + .body("data.directDownload", CoreMatchers.notNullValue()) + .statusCode(OK.getStatusCode()); + + // Test dataset under root with default storage driver + Response getStorageDriverResponse = UtilIT.getStorageDriver("root", apiToken, Boolean.TRUE); + getStorageDriverResponse.prettyPrint(); + data = JsonUtil.getJsonObject(getStorageDriverResponse.getBody().asString()); + name = data.getJsonObject("data").getString("name"); + String type = data.getJsonObject("data").getString("type"); + String label = data.getJsonObject("data").getString("label"); + createDataset = UtilIT.createRandomDatasetViaNativeApi("root", apiToken); + datasetId = UtilIT.getDatasetIdFromResponse(createDataset); + getDriver = UtilIT.getDatasetStorageDriver(datasetId, apiToken); + getDriver.prettyPrint(); + assertEquals(200, getDriver.getStatusCode()); + getDriver.then().assertThat() + .body("data.name", CoreMatchers.equalTo(name)) + .body("data.type", CoreMatchers.equalTo(type)) + .body("data.label", CoreMatchers.equalTo(label)) + .body("data.directUpload", CoreMatchers.notNullValue()) + .body("data.directDownload", CoreMatchers.notNullValue()) + .statusCode(OK.getStatusCode()); + } + @Test public void testGetCanDownloadAtLeastOneFile() { Response createUserResponse = UtilIT.createRandomUser(); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java index 4f9fbae7ffc..eb247ce6c7f 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java @@ -1,5 +1,6 @@ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.util.json.JsonParseException; import edu.harvard.iq.dataverse.util.json.JsonParser; import edu.harvard.iq.dataverse.util.json.JsonUtil; @@ -45,6 +46,7 @@ import io.restassured.path.json.JsonPath; import org.hamcrest.CoreMatchers; import org.hamcrest.Matchers; +import static org.hamcrest.Matchers.greaterThan; public class DataversesIT { @@ -677,6 +679,235 @@ public void testImportDDI() throws IOException, InterruptedException { Response deleteUserResponse = UtilIT.deleteUser(username); assertEquals(200, deleteUserResponse.getStatusCode()); } + + @Test + public void testGetLinkableDataverses(){ + Response createUser = UtilIT.createRandomUser(); + String apiToken = UtilIT.getApiTokenFromResponse(createUser); + String username = UtilIT.getUsernameFromResponse(createUser); + + Response createDataverseResponse = UtilIT.createRandomDataverse(apiToken); + String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse); + + //Create dataverse for linking + Response createDataverseResponseForLinking = UtilIT.createRandomDataverse(apiToken); + String dataverseAliasForLinking = UtilIT.getAliasFromResponse(createDataverseResponseForLinking); + + + Response publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAlias, apiToken); + assertEquals(200, publishDataverse.getStatusCode()); + + publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAliasForLinking, apiToken); + assertEquals(200, publishDataverse.getStatusCode()); + + Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); + createDatasetResponse.prettyPrint(); + String datasetPersistentId = UtilIT.getDatasetPersistentIdFromResponse(createDatasetResponse); + + Integer datasetId = UtilIT.getDatasetIdFromResponse(createDatasetResponse); + UtilIT.publishDatasetViaNativeApi(datasetPersistentId, "major", apiToken); + + Response getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasForLinking); + getLinkableDataverses.prettyPrint(); + getLinkableDataverses.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[0].alias", equalTo(dataverseAliasForLinking)); + + Response getLinkableDataversesForDataverse = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, dataverseAliasForLinking); + getLinkableDataversesForDataverse.prettyPrint(); + getLinkableDataverses.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[0].alias", equalTo(dataverseAliasForLinking)); + + //Should be able to get based on a partial alias... + // Partial must include the first part of the name + String searchTerm = dataverseAliasForLinking.substring(0, 7); + + Response getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, searchTerm); + getLinkableDataversesForDataversePartial.prettyPrint(); + getLinkableDataversesForDataversePartial.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[0].alias", equalTo(dataverseAliasForLinking)); + + //if I give a blank search term i should get the one that I have perms on + Response getLinkableDataversesForDataverseBlank = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, ""); + getLinkableDataversesForDataverseBlank.prettyPrint(); + getLinkableDataversesForDataverseBlank.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[0].alias", equalTo(dataverseAliasForLinking)); + + + //Try with empty string search term + searchTerm = ""; + getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, searchTerm); + getLinkableDataversesForDataversePartial.prettyPrint(); + getLinkableDataversesForDataversePartial.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[0].alias", equalTo(dataverseAliasForLinking)); + + //Should get same result if search term is null + getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, null); + getLinkableDataversesForDataversePartial.prettyPrint(); + getLinkableDataversesForDataversePartial.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[0].alias", equalTo(dataverseAliasForLinking)); + + // Create a child of the linking DV + Response createLevel1a = UtilIT.createSubDataverse(UtilIT.getRandomDvAlias() + "-level1a", null, apiToken, dataverseAliasForLinking); + createLevel1a.prettyPrint(); + String childLinking = UtilIT.getAliasFromResponse(createLevel1a); + + //Try to link the child - should not link to parent - should link to "uncle" + getLinkableDataversesForDataversePartial = UtilIT.getLinkableDataverses("dataverse", childLinking, apiToken, searchTerm); + getLinkableDataversesForDataversePartial.prettyPrint(); + getLinkableDataversesForDataversePartial.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data[0].alias", equalTo(dataverseAlias)); + + //Try with bad target alias + searchTerm = ""; + Response getLinkableDataversesForDataverseBadId = UtilIT.getLinkableDataverses("dataverse", "junque@#", apiToken, searchTerm); + getLinkableDataversesForDataverseBadId.prettyPrint(); + getLinkableDataversesForDataverseBadId.then().assertThat() + .statusCode(NOT_FOUND.getStatusCode()); + + // create new user and dataverse - the new dataverse should not be available to the first user for linking... + Response createUserTwo = UtilIT.createRandomUser(); + String apiTokenTwo = UtilIT.getApiTokenFromResponse(createUserTwo); + String usernameTwo = UtilIT.getUsernameFromResponse(createUserTwo); + + //Create dataverse that should be unavailable for linking + Response createDataverseResponseUnavailableForLinking = UtilIT.createRandomDataverse(apiTokenTwo); + createDataverseResponseUnavailableForLinking.prettyPrint(); + String dataverseAliasUnavailableForLinking = UtilIT.getAliasFromResponse(createDataverseResponseUnavailableForLinking); + publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAliasUnavailableForLinking, apiTokenTwo); + publishDataverse.prettyPrint(); + assertEquals(200, publishDataverse.getStatusCode()); + + //user 3 will not have permissions + Response createUserThree = UtilIT.createRandomUser(); + String apiTokenThree = UtilIT.getApiTokenFromResponse(createUserThree); + String usernameThree = UtilIT.getUsernameFromResponse(createUserThree); + + Response getUnavailableForDatasetBadSearchTerm = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, "xxQQQ-Batman"); + getUnavailableForDatasetBadSearchTerm.prettyPrint(); + getUnavailableForDatasetBadSearchTerm.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(0)); + + Response getUnavailableForDataset = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasUnavailableForLinking); + getUnavailableForDataset.prettyPrint(); + getUnavailableForDataset.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(0)); + + Response getGuestUnavailableForDataset = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, "", dataverseAliasUnavailableForLinking); + getGuestUnavailableForDataset.prettyPrint(); + getGuestUnavailableForDataset.then().assertThat() + .statusCode(FORBIDDEN.getStatusCode()); + + Response getUnavailableForDataverse = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiToken, dataverseAliasUnavailableForLinking); + getUnavailableForDataverse.prettyPrint(); + getUnavailableForDataverse.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(0)); + + Response getNoPermsOnAnyCollection = UtilIT.getLinkableDataverses("dataverse", dataverseAlias, apiTokenThree, ""); + getNoPermsOnAnyCollection.prettyPrint(); + getNoPermsOnAnyCollection.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(0)); + + //now link a dataset and see that it's unavailable in the future + + Response makeSuperUser = UtilIT.setSuperuserStatus(username, Boolean.TRUE); + + Response linkDataset = UtilIT.linkDataset(datasetPersistentId, dataverseAliasForLinking, apiToken); + linkDataset.prettyPrint(); + linkDataset.then().assertThat() + .statusCode(OK.getStatusCode()); + + //set user api back to non-super user so perms are limited + UtilIT.setSuperuserStatus(username, Boolean.FALSE); + + //should get an empty list because dataset is already linked + getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasForLinking); + getLinkableDataverses.prettyPrint(); + getLinkableDataverses.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(0)); + + //if you ask for already linked you should get one + getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasForLinking, true); + + getLinkableDataverses.prettyPrint(); + getLinkableDataverses.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(1)); + + //if you ask for already linked you should get one unless you don't have perms + getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiTokenThree, dataverseAliasForLinking, true); + + getLinkableDataverses.prettyPrint(); + getLinkableDataverses.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(0)); + + //if you ask for already linked you should get one unless there's a bad search terms + getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, "QQQBatmanSymbol", true); + + getLinkableDataverses.prettyPrint(); + getLinkableDataverses.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(0)); + + //set user api back to super user for cleanup + UtilIT.setSuperuserStatus(username, Boolean.TRUE); + + //Create dataverse that should be available for super user + Response createDataverseResponseAvailableForLinkingBySuperUser = UtilIT.createRandomDataverse(apiTokenTwo); + createDataverseResponseAvailableForLinkingBySuperUser.prettyPrint(); + String dataverseAliasAvailableForLinkingBySuperUser = UtilIT.getAliasFromResponse(createDataverseResponseAvailableForLinkingBySuperUser); + publishDataverse = UtilIT.publishDataverseViaNativeApi(dataverseAliasAvailableForLinkingBySuperUser, apiToken); + publishDataverse.prettyPrint(); + assertEquals(200, publishDataverse.getStatusCode()); + + //First see that the super user gets dataverses to link to + getLinkableDataverses = UtilIT.getLinkableDataverses("dataset", datasetPersistentId, apiToken, dataverseAliasAvailableForLinkingBySuperUser); + getLinkableDataverses.prettyPrint(); + getLinkableDataverses.then().assertThat() + .statusCode(OK.getStatusCode()) + .body("data.size()", equalTo(1)); + + // Clean up + Response destroyDatasetResponse = UtilIT.destroyDataset(datasetId, apiToken); + assertEquals(200, destroyDatasetResponse.getStatusCode()); + + Response deleteDataverseResponseSuper = UtilIT.deleteDataverse(dataverseAliasAvailableForLinkingBySuperUser, apiToken); + assertEquals(200, deleteDataverseResponseSuper.getStatusCode()); + + Response deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAlias, apiToken); + assertEquals(200, deleteDataverseResponse.getStatusCode()); + + deleteDataverseResponse = UtilIT.deleteDataverse(childLinking, apiToken); + assertEquals(200, deleteDataverseResponse.getStatusCode()); + + deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAliasForLinking, apiToken); + assertEquals(200, deleteDataverseResponse.getStatusCode()); + + deleteDataverseResponse = UtilIT.deleteDataverse(dataverseAliasUnavailableForLinking, apiToken); + assertEquals(200, deleteDataverseResponse.getStatusCode()); + + Response deleteUserResponse = UtilIT.deleteUser(usernameTwo); + assertEquals(200, deleteUserResponse.getStatusCode()); + + deleteUserResponse = UtilIT.deleteUser(usernameThree); + assertEquals(200, deleteUserResponse.getStatusCode()); + + deleteUserResponse = UtilIT.deleteUser(username); + assertEquals(200, deleteUserResponse.getStatusCode()); + + } @Test public void testImport() throws IOException, InterruptedException { @@ -1255,6 +1486,77 @@ public void testUpdateInputLevels() { updateDataverseInputLevelsResponse.then().assertThat() .body("message", equalTo("Error while updating dataverse input levels: Input level list cannot be null or empty")) .statusCode(INTERNAL_SERVER_ERROR.getStatusCode()); + + //Add new types and see that previously changed ones remain as before... #11387 + testInputLevelNames = new String[]{"subtitle", "relatedMaterial"}; + + testRequiredInputLevels = new boolean[] {false, false}; + testIncludedInputLevels = new boolean[] {true, true}; + boolean [] testDisplayOnCreate = new boolean[] {true, false}; + updateDataverseInputLevelsResponse = UtilIT.updateDataverseInputLevels(dataverseAlias, testInputLevelNames, testRequiredInputLevels, testIncludedInputLevels, testDisplayOnCreate, apiToken); + updateDataverseInputLevelsResponse.prettyPrint(); + int subtitleInputLevelIndex = -1; + int relatedMaterialInputLevelIndex = -1; + int i = 0; + + while (updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)) != null){ + actualInputLevelName = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)).toString(); + if (actualInputLevelName.equals("subtitle")){ + subtitleInputLevelIndex = i; + } + if (actualInputLevelName.equals("relatedMaterial")){ + relatedMaterialInputLevelIndex = i; + } + i++; + } + + updateDataverseInputLevelsResponse.then().assertThat() + .body(String.format("data.inputLevels[%d].include", subtitleInputLevelIndex), equalTo(true)) + .body(String.format("data.inputLevels[%d].required", subtitleInputLevelIndex), equalTo(false)) + .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleInputLevelIndex), equalTo(true)) + .body(String.format("data.inputLevels[%d].include", relatedMaterialInputLevelIndex), equalTo(true)) + .body(String.format("data.inputLevels[%d].required", relatedMaterialInputLevelIndex), equalTo(false)) + .body(String.format("data.inputLevels[%d].displayOnCreate", relatedMaterialInputLevelIndex), equalTo(false)) + .statusCode(OK.getStatusCode()); + + actualFieldTypeName1 = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", subtitleInputLevelIndex)); + actualFieldTypeName2 = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", relatedMaterialInputLevelIndex)); + assertNotEquals(actualFieldTypeName1, actualFieldTypeName2); + assertThat(testInputLevelNames, hasItemInArray(actualFieldTypeName1)); + assertThat(testInputLevelNames, hasItemInArray(actualFieldTypeName2)); + + + testInputLevelNames = new String[]{"subtitle", "otherReferences"}; + testRequiredInputLevels = new boolean[] {false, false}; + testIncludedInputLevels = new boolean[] {true, true}; + testDisplayOnCreate = new boolean[] {false, true}; + + updateDataverseInputLevelsResponse = UtilIT.updateDataverseInputLevels(dataverseAlias, testInputLevelNames, testRequiredInputLevels, testIncludedInputLevels, testDisplayOnCreate, apiToken); + updateDataverseInputLevelsResponse.prettyPrint(); + + subtitleInputLevelIndex = 0; + i = 0; + + while (updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)) != null) { + actualInputLevelName = updateDataverseInputLevelsResponse.then().extract().path(String.format("data.inputLevels[%d].datasetFieldTypeName", i)).toString(); + if (actualInputLevelName.equals("subtitle")) { + subtitleInputLevelIndex = i; + } + i++; + } + + //make sure subtitle got changed to false + updateDataverseInputLevelsResponse.then().assertThat() + .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleInputLevelIndex), equalTo(false)) + .statusCode(OK.getStatusCode()); + + //make superuser for cleanup + String username = UtilIT.getUsernameFromResponse(createUserResponse); + UtilIT.setSuperuserStatus(username, Boolean.TRUE); + Response deleteDataverse1Response = UtilIT.deleteDataverse(dataverseAlias, apiToken); + deleteDataverse1Response.prettyPrint(); + assertEquals(200, deleteDataverse1Response.getStatusCode()); + } @Test @@ -2256,12 +2558,17 @@ public void testUpdateInputLevelDisplayOnCreateOverride() { .body("data.inputLevels[0].displayOnCreate", equalTo(true)) .body("data.inputLevels[0].datasetFieldTypeName", equalTo("notesText")); + updateResponse = UtilIT.updateDataverseInputLevelDisplayOnCreate( dataverseAlias, "subtitle", true, apiToken); + + String actualInputLevelName = updateResponse.then().extract().path("data.inputLevels[0].datasetFieldTypeName"); + int subtitleInputLevelIndex = actualInputLevelName.equals("subtitle") ? 0 : 1; + updateResponse.prettyPrint(); updateResponse.then().assertThat() .statusCode(OK.getStatusCode()) - .body("data.inputLevels[0].displayOnCreate", equalTo(true)) - .body("data.inputLevels[0].datasetFieldTypeName", equalTo("subtitle")); + .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleInputLevelIndex), equalTo(true)) + .body(String.format("data.inputLevels[%d].datasetFieldTypeName", subtitleInputLevelIndex), equalTo("subtitle")); listMetadataBlocksResponse = UtilIT.listMetadataBlocks(dataverseAlias, true, true, apiToken); listMetadataBlocksResponse.prettyPrint(); @@ -2274,28 +2581,51 @@ public void testUpdateInputLevelDisplayOnCreateOverride() { .body("data[0].displayName", equalTo("Citation Metadata")) .body("data.size()", equalTo(expectedOnlyDisplayedOnCreateNumberOfMetadataBlocks)) .body("data[0].fields.author.childFields.size()", is(4)); - - updateResponse = UtilIT.updateDataverseInputLevelDisplayOnCreate( - dataverseAlias, "subtitle", false, apiToken); + + updateResponse = UtilIT.updateDataverseInputLevelDisplayOnCreate(dataverseAlias, "subtitle", false, apiToken); + actualInputLevelName = updateResponse.then().extract().path("data.inputLevels[0].datasetFieldTypeName"); + int subtitleIndex = actualInputLevelName.equals("subtitle") ? 0 : 1; updateResponse.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.inputLevels[0].displayOnCreate", equalTo(false)) - .body("data.inputLevels[0].datasetFieldTypeName", equalTo("subtitle")); - + .body(String.format("data.inputLevels[%d].displayOnCreate", subtitleIndex), equalTo(false)) + .body(String.format("data.inputLevels[%d].datasetFieldTypeName", subtitleIndex), equalTo("subtitle")) + .statusCode(OK.getStatusCode()); + } @Test - public void testCreateAndGetTemplates() { + public void testCreateAndGetTemplates() throws JsonParseException { Response createUserResponse = UtilIT.createRandomUser(); String apiToken = UtilIT.getApiTokenFromResponse(createUserResponse); Response createSecondUserResponse = UtilIT.createRandomUser(); String secondApiToken = UtilIT.getApiTokenFromResponse(createSecondUserResponse); - + + /* + We need to make this a non-inherited metadatablocks so the get template will only get templates from current dv + */ + Response createDataverseResponse = UtilIT.createRandomDataverse(apiToken); createDataverseResponse.then().assertThat().statusCode(CREATED.getStatusCode()); String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse); + String newName = "New Test Dataverse Name"; + String newAffiliation = "New Test Dataverse Affiliation"; + String newDataverseType = Dataverse.DataverseType.TEACHING_COURSES.toString(); + String[] newContactEmails = new String[]{"new_email@dataverse.com"}; + String[] newInputLevelNames = new String[]{"geographicCoverage"}; + String[] newFacetIds = new String[]{"contributorName"}; + String[] newMetadataBlockNames = new String[]{"citation", "geospatial", "biomedical"}; + + //Giving the new Dataverse updated metadatablocks so that it will not inherit templates + Response updateDataverseResponse = UtilIT.updateDataverse( + dataverseAlias, dataverseAlias, newName, newAffiliation, newDataverseType, newContactEmails, newInputLevelNames, + null, newMetadataBlockNames, apiToken, + Boolean.FALSE, Boolean.FALSE, null + ); + + updateDataverseResponse.then().assertThat() + .statusCode(OK.getStatusCode()); + // Create a template String jsonString = """ @@ -2333,6 +2663,7 @@ public void testCreateAndGetTemplates() { jsonString, apiToken ); + createTemplateResponse.then().assertThat().statusCode(OK.getStatusCode()) .body("data.name", equalTo("Dataverse template")) .body("data.isDefault", equalTo(true)) @@ -2373,6 +2704,26 @@ public void testCreateAndGetTemplates() { getTemplateResponse.then().assertThat().statusCode(UNAUTHORIZED.getStatusCode()); } + @Test + public void testGetStorageDriver() { + Response updatedStorageDriver = UtilIT.getStorageDriver("root", getSuperuserToken(), Boolean.TRUE); + updatedStorageDriver.prettyPrint(); + updatedStorageDriver.then().assertThat() + .body("data.name", CoreMatchers.notNullValue()) + .body("data.type", CoreMatchers.notNullValue()) + .body("data.label", CoreMatchers.notNullValue()) + .body("data.directUpload", CoreMatchers.nullValue()) + .body("data.directDownload", CoreMatchers.nullValue()) + .statusCode(200); + + // Root without default is undefined + updatedStorageDriver = UtilIT.getStorageDriver("root", getSuperuserToken(), null); + updatedStorageDriver.prettyPrint(); + updatedStorageDriver.then().assertThat() + .body("data.name", CoreMatchers.equalTo(DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER)) + .statusCode(200); + } + private String getSuperuserToken() { Response createResponse = UtilIT.createRandomUser(); String adminApiToken = UtilIT.getApiTokenFromResponse(createResponse); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java b/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java index 137c2ef4c7b..24625c87ce2 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/S3AccessIT.java @@ -1,5 +1,6 @@ package edu.harvard.iq.dataverse.api; +import org.hamcrest.CoreMatchers; import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; import software.amazon.awssdk.core.ResponseInputStream; @@ -142,7 +143,7 @@ public void testNonDirectUpload() { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.message", equalTo("undefined")) + .body("data.name", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); @@ -153,6 +154,9 @@ public void testNonDirectUpload() { Response updatedStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); updatedStorageDriver.prettyPrint(); updatedStorageDriver.then().assertThat() + .body("data.type", CoreMatchers.notNullValue()) + .body("data.label", CoreMatchers.notNullValue()) + .body("data.directUpload", CoreMatchers.nullValue()) .statusCode(200); Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken); @@ -273,7 +277,7 @@ public void testDirectUpload() { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.message", equalTo("undefined")) + .body("data.name", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); @@ -491,7 +495,7 @@ public void testDirectUploadDetectStataFile() { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.message", equalTo("undefined")) + .body("data.name", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); @@ -689,7 +693,7 @@ public void testDirectUploadWithFileCountLimit() throws JsonParseException { Response originalStorageDriver = UtilIT.getStorageDriver(dataverseAlias, superuserApiToken); originalStorageDriver.prettyPrint(); originalStorageDriver.then().assertThat() - .body("data.message", equalTo("undefined")) + .body("data.name", equalTo("undefined")) .statusCode(200); Response setStorageDriverToS3 = UtilIT.setStorageDriver(dataverseAlias, driverLabel, superuserApiToken); diff --git a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java index eba8181e566..c11f66aa749 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/UtilIT.java @@ -2850,9 +2850,13 @@ static Response listStorageDrivers(String apiToken) { } static Response getStorageDriver(String dvAlias, String apiToken) { + return getStorageDriver(dvAlias, apiToken, null); + } + static Response getStorageDriver(String dvAlias, String apiToken, Boolean getEffective) { + String params = getEffective != null ? "?getEffective=" + getEffective : ""; return given() .header(API_TOKEN_HTTP_HEADER, apiToken) - .get("/api/admin/dataverse/" + dvAlias + "/storageDriver"); + .get("/api/admin/dataverse/" + dvAlias + "/storageDriver" + params); } static Response setStorageDriver(String dvAlias, String label, String apiToken) { @@ -4460,6 +4464,12 @@ static Response setDatasetStorageDriver(Integer datasetId, String driverLabel, S .put("/api/datasets/" + datasetId + "/storageDriver"); } + static Response getDatasetStorageDriver(Integer datasetId, String apiToken) { + return given() + .header(API_TOKEN_HTTP_HEADER, apiToken) + .get("/api/datasets/" + datasetId + "/storageDriver"); + } + /** GET on /api/admin/savedsearches/list */ static Response getSavedSearchList() { return given().get("/api/admin/savedsearches/list"); @@ -4533,6 +4543,18 @@ public static Response updateDataverseInputLevels(String dataverseAlias, String[ .contentType(ContentType.JSON) .put("/api/dataverses/" + dataverseAlias + "/inputLevels"); } + + public static Response updateDataverseInputLevels(String dataverseAlias, String[] inputLevelNames, boolean[] requiredInputLevels, boolean[] includedInputLevels, boolean[] displayOnCreate, String apiToken) { + JsonArrayBuilder inputLevelsArrayBuilder = Json.createArrayBuilder(); + for (int i = 0; i < inputLevelNames.length; i++) { + inputLevelsArrayBuilder.add(createInputLevelObject(inputLevelNames[i], requiredInputLevels[i], includedInputLevels[i], displayOnCreate[i])); + } + return given() + .header(API_TOKEN_HTTP_HEADER, apiToken) + .body(inputLevelsArrayBuilder.build().toString()) + .contentType(ContentType.JSON) + .put("/api/dataverses/" + dataverseAlias + "/inputLevels"); + } private static JsonObjectBuilder createInputLevelObject(String name, boolean required, boolean include) { return Json.createObjectBuilder() @@ -4540,6 +4562,14 @@ private static JsonObjectBuilder createInputLevelObject(String name, boolean req .add("required", required) .add("include", include); } + + private static JsonObjectBuilder createInputLevelObject(String name, boolean required, boolean include, boolean displayOnCreate) { + return Json.createObjectBuilder() + .add("datasetFieldTypeName", name) + .add("required", required) + .add("include", include) + .add("displayOnCreate", displayOnCreate); + } public static Response getOpenAPI(String accept, String format) { Response response = given() @@ -4728,6 +4758,59 @@ static Response updateDataverseFeaturedItem(long featuredItemId, String apiToken) { return updateDataverseFeaturedItem(featuredItemId, content, displayOrder, keepFile, pathToFile, null, null, apiToken); } + + static Response getLinkableDataverses(String type, String dvObjectId, String apiToken, String searchTerm) { + return getLinkableDataverses(type, dvObjectId, apiToken, searchTerm, false); + } + + static Response getLinkableDataverses(String type, String dvObjectId, String apiToken, String searchTerm, boolean alreadyLinked) { + + String idInPath = dvObjectId; // Assume it's a number to start. + String optionalQueryParam = ""; // If idOrPersistentId is a number we'll just put it in the path. + if (type.equals("dataset")) { + if (!NumberUtils.isCreatable(idInPath)) { + idInPath = ":persistentId"; + if (searchTerm == null || searchTerm.isEmpty() ) { + optionalQueryParam = "?persistentId=" + dvObjectId; + } else { + optionalQueryParam = "&persistentId=" + dvObjectId; + } + } + } + + if (alreadyLinked){ + if (optionalQueryParam.isEmpty() && (searchTerm == null || searchTerm.isEmpty())){ + optionalQueryParam = "?alreadyLinking=true"; + } else { + optionalQueryParam = optionalQueryParam + "&alreadyLinking=true"; + } + } + + if (searchTerm == null) { + if (!apiToken.isEmpty()) { + return given() + .header(API_TOKEN_HTTP_HEADER, apiToken) + .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses" + optionalQueryParam); + + } else { + return given() + .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses" + optionalQueryParam); + } + + } else { + if (!apiToken.isEmpty()) { + return given() + .header(API_TOKEN_HTTP_HEADER, apiToken) + .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses?searchTerm=" + searchTerm + optionalQueryParam); + + } else { + return given() + .get("/api/dataverses/" + idInPath + "/" + type + "/linkingDataverses?searchTerm=" + searchTerm + optionalQueryParam); + } + + } + } + static Response updateDataverseFeaturedItem(long featuredItemId, String content, int displayOrder, diff --git a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java index 73880b78e7b..2d2fd943d4d 100644 --- a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/CreateDataverseCommandTest.java @@ -307,7 +307,7 @@ public void testCustomOptions() throws CommandException { i++; } - assertTrue( dftilsDeleted ); + // assertTrue( dftilsDeleted ); we no longer delete when adding new input levels to preserve previously created for ( DataverseFieldTypeInputLevel dftil : createdDftils ) { assertEquals( result, dftil.getDataverse() ); } diff --git a/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java new file mode 100644 index 00000000000..7f167370c63 --- /dev/null +++ b/src/test/java/edu/harvard/iq/dataverse/engine/command/impl/GetLinkingDataverseListCommandTest.java @@ -0,0 +1,163 @@ +package edu.harvard.iq.dataverse.engine.command.impl; + +import edu.harvard.iq.dataverse.*; +import edu.harvard.iq.dataverse.authorization.Permission; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.engine.command.CommandContext; +import edu.harvard.iq.dataverse.engine.command.DataverseRequest; +import edu.harvard.iq.dataverse.engine.command.exception.CommandException; +import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; +import edu.harvard.iq.dataverse.util.BundleUtil; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@ExtendWith(MockitoExtension.class) +public class GetLinkingDataverseListCommandTest { + + @Mock + private DataverseRequest dataverseRequest; + @Mock + private CommandContext commandContext; + @Mock + private PermissionServiceBean permissionService; + @Mock + private DataverseServiceBean dataverseService; + @Mock + private AuthenticatedUser authenticatedUser; + + @BeforeEach + public void setUp() { + Mockito.when(dataverseRequest.getUser()).thenReturn(authenticatedUser); + } + + @Test + public void execute_shouldThrowException_whenUserIsNotAuthenticated() { + // Arrange + Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(false); + DvObject dvObject = new Dataset(); + GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, dvObject, ""); + + // Act & Assert + assertThrows(IllegalCommandException.class, () -> { + sut.execute(commandContext); + }, BundleUtil.getStringFromBundle("dataverse.link.user")); + } + + @Test + public void execute_shouldReturnPermittedList_forLinkingDataset() throws CommandException { + // Arrange + String searchTerm = "test"; + Dataset datasetToLink = new Dataset(); + List permittedList = Arrays.asList(new Dataverse(), new Dataverse()); + + Mockito.when(commandContext.permissions()).thenReturn(permissionService); + Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); + Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); + Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, searchTerm)) + .thenReturn(permittedList); + Mockito.when(dataverseService.removeUnlinkableDataverses(permittedList, datasetToLink)) + .thenReturn(permittedList); // Assume none are removed for this test + + GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, datasetToLink, searchTerm); + + // Act + List result = sut.execute(commandContext); + + // Assert + assertEquals(2, result.size()); + assertEquals(permittedList, result); + Mockito.verify(permissionService).findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, searchTerm); + Mockito.verify(dataverseService).removeUnlinkableDataverses(permittedList, datasetToLink); + } + + @Test + public void execute_shouldReturnPermittedList_forLinkingDataverse() throws CommandException { + // Arrange + String searchTerm = "test"; + Dataverse dataverseToLink = new Dataverse(); + List permittedList = Collections.singletonList(new Dataverse()); + + Mockito.when(commandContext.permissions()).thenReturn(permissionService); + Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); + Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); + Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataverse, searchTerm)) + .thenReturn(permittedList); + Mockito.when(dataverseService.removeUnlinkableDataverses(permittedList, dataverseToLink)) + .thenReturn(permittedList); + + GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, dataverseToLink, searchTerm); + + // Act + List result = sut.execute(commandContext); + + // Assert + assertEquals(1, result.size()); + assertEquals(permittedList, result); + Mockito.verify(permissionService).findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataverse, searchTerm); + Mockito.verify(dataverseService).removeUnlinkableDataverses(permittedList, dataverseToLink); + } + + @Test + public void execute_shouldUseEmptyString_whenSearchTermIsNull() throws CommandException { + // Arrange + String searchTerm = null; + Dataset datasetToLink = new Dataset(); + List expectedList = Collections.emptyList(); + + Mockito.when(commandContext.permissions()).thenReturn(permissionService); + // if there are none found then the remove unlinkable doesn't get run + // Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); + Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); + // Note: verify that an empty string "" is passed instead of null + Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, "")) + .thenReturn(expectedList); + // if there are none found then the remove unlinkable doesn't get run + // Mockito.when(dataverseService.removeUnlinkableDataverses(expectedList, datasetToLink)).thenReturn(expectedList); + + GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, datasetToLink, searchTerm); + + // Act + List result = sut.execute(commandContext); + + // Assert + assertTrue(result.isEmpty()); + Mockito.verify(permissionService).findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, ""); + } + + @Test + public void execute_shouldReturnEmptyList_whenNoPermittedCollectionsFound() throws CommandException { + // Arrange + String searchTerm = "nonexistent"; + Dataset datasetToLink = new Dataset(); + List emptyList = Collections.emptyList(); + + Mockito.when(commandContext.permissions()).thenReturn(permissionService); + // if there are none found then the remove unlinkable doesn't get run + // Mockito.when(commandContext.dataverses()).thenReturn(dataverseService); + Mockito.when(authenticatedUser.isAuthenticated()).thenReturn(true); + Mockito.when(permissionService.findPermittedCollections(dataverseRequest, authenticatedUser, Permission.LinkDataset, searchTerm)) + .thenReturn(emptyList); + // if there are none found then the remove unlinkable doesn't get run + // Mockito.when(dataverseService.removeUnlinkableDataverses(emptyList, datasetToLink)).thenReturn(emptyList); + + GetLinkingDataverseListCommand sut = new GetLinkingDataverseListCommand(dataverseRequest, datasetToLink, searchTerm); + + // Act + List result = sut.execute(commandContext); + + // Assert + assertTrue(result.isEmpty()); + } +} From cf20973ea5d9a53f47187a9838bf1f22a3624197 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 23 Oct 2025 17:01:36 -0400 Subject: [PATCH 061/578] addressing comments --- .../command/impl/MoveDatasetCommand.java | 47 ++++++++++++------- .../edu/harvard/iq/dataverse/api/MoveIT.java | 47 +++++++++---------- 2 files changed, 51 insertions(+), 43 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java index 513e936dabf..335a28d45b2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java @@ -32,8 +32,8 @@ * @author skraffmi */ @RequiredPermissionsMap({ - @RequiredPermissions(dataverseName = "moved", value = {Permission.PublishDataset}) - , @RequiredPermissions(dataverseName = "destination", value = {Permission.AddDataset, Permission.PublishDataset}) + @RequiredPermissions(dataverseName = "moved", value = {Permission.PublishDataset}) + , @RequiredPermissions(dataverseName = "destination", value = {Permission.AddDataset, Permission.PublishDataset}) }) public class MoveDatasetCommand extends AbstractVoidCommand { @@ -42,10 +42,11 @@ public class MoveDatasetCommand extends AbstractVoidCommand { final Dataset moved; final Dataverse destination; final Boolean force; - private boolean allowSelfNotification = false; + final Boolean allowSelfNotification; + final Dataverse originalOwner; public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force) { - this( aRequest, moved, destination, force, Boolean.FALSE); + this( aRequest, moved, destination, force, null); } public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse destination, Boolean force, Boolean allowSelfNotification) { super( @@ -57,6 +58,7 @@ public MoveDatasetCommand(DataverseRequest aRequest, Dataset moved, Dataverse de this.destination = destination; this.force= force; this.allowSelfNotification = allowSelfNotification; + this.originalOwner = moved.getOwner(); } @Override @@ -75,13 +77,13 @@ public void executeImpl(CommandContext ctxt) throws CommandException { if (moved.getOwner().equals(destination)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dashboard.move.dataset.command.error.targetDataverseSameAsOriginalDataverse"), this); } - + // if dataset is published make sure that its target is published - + if (moved.isReleased() && !destination.isReleased()){ throw new IllegalCommandException(BundleUtil.getStringFromBundle("dashboard.move.dataset.command.error.targetDataverseUnpublishedDatasetPublished", Arrays.asList(destination.getDisplayName())), this); } - + //if the datasets guestbook is not contained in the new dataverse then remove it if (moved.getGuestbook() != null) { Guestbook gb = moved.getGuestbook(); @@ -100,15 +102,15 @@ public void executeImpl(CommandContext ctxt) throws CommandException { } } } - + // generate list of all possible parent dataverses to check against List ownersToCheck = new ArrayList<>(); ownersToCheck.add(destination); if (destination.getOwners() != null) { ownersToCheck.addAll(destination.getOwners()); } - - // if the dataset is linked to the new dataverse or any of + + // if the dataset is linked to the new dataverse or any of // its parent dataverses then remove the link List linkingDatasets = new ArrayList<>(); if (moved.getDatasetLinkingDataverses() != null) { @@ -127,7 +129,7 @@ public void executeImpl(CommandContext ctxt) throws CommandException { } } } - + if (removeGuestbook || removeLinkDs) { StringBuilder errorString = new StringBuilder(); if (removeGuestbook) { @@ -138,25 +140,30 @@ public void executeImpl(CommandContext ctxt) throws CommandException { } throw new UnforcedCommandException(errorString.toString(), this); } - + // 6575 if dataset is submitted for review and the default contributor // role includes dataset publish then remove the lock - + if (moved.isLockedFor(DatasetLock.Reason.InReview) && destination.getDefaultContributorRole().permissions().contains(Permission.PublishDataset)) { ctxt.datasets().removeDatasetLocks(moved, DatasetLock.Reason.InReview); } // OK, move - Dataverse originalOwner = moved.getOwner(); moved.setOwner(destination); ctxt.em().merge(moved); - sendNotification(moved, originalOwner, ctxt); boolean doNormalSolrDocCleanUp = true; ctxt.index().asyncIndexDataset(moved, doNormalSolrDocCleanUp); } + + @Override + public boolean onSuccess(CommandContext ctxt, Object r) { + sendNotification(moved, originalOwner, ctxt); + return true; + } + /** * Sends notifications to those able to publish the dataset upon the successful move of a dataset. *

          @@ -181,14 +188,17 @@ protected void sendNotification(Dataset dataset, Dataverse originalOwner, Comman // 3. Get all users with publish permission on the dataset's original owner (dataverse) and notify them. Map recipients = ctxt.permissions().getDistinctUsersWithPermissionOn(Permission.PublishDataset, originalOwner); - // make sure the requestor is in the recipient list in case they don't match the permission + // make sure the requestor is in the recipient list in case they don't match the permission but only if allowSelfNotification is true if (requestor != null) { - recipients.put(requestor.getIdentifier(), requestor); + if (Boolean.TRUE.equals(allowSelfNotification)) { + recipients.put(requestor.getIdentifier(), requestor); + } else { + recipients.remove(requestor.getIdentifier()); + } } recipients.values() .stream() - .filter(recipient -> allowSelfNotification || !recipient.equals(requestor)) .forEach(recipient -> ctxt.notifications().sendNotification( recipient, Timestamp.from(Instant.now()), @@ -200,3 +210,4 @@ protected void sendNotification(Dataset dataset, Dataverse originalOwner, Comman )); } } + diff --git a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java index a547c44eea2..1c3a64bbf51 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/MoveIT.java @@ -23,6 +23,7 @@ import static org.hamcrest.CoreMatchers.*; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -68,7 +69,7 @@ public void testMoveDataset() { Response noPermToCreateDataset = UtilIT.createRandomDatasetViaNativeApi(curatorDataverseAlias1, authorApiToken); noPermToCreateDataset.prettyPrint(); noPermToCreateDataset.then().assertThat().statusCode(UNAUTHORIZED.getStatusCode()); - + Response grantAuthorAddDataset = UtilIT.grantRoleOnDataverse(curatorDataverseAlias1, DataverseRole.DS_CONTRIBUTOR.toString(), "@" + authorUsername, curatorApiToken); grantAuthorAddDataset.prettyPrint(); grantAuthorAddDataset.then().assertThat() @@ -195,10 +196,6 @@ public void testMoveDatasetNotification() { .statusCode(CREATED.getStatusCode()); Integer datasetId = UtilIT.getDatasetIdFromResponse(createDataset); - // clear existing notifications (so the DATASETMOVED notification will be the only one) - clearNotifications(user1ApiToken); - clearNotifications(user2ApiToken); - // User1(superuser) moves the dataset from dataverse2 to dataverse1 Response moveDataset = UtilIT.moveDataset(datasetId.toString(), dataverseAlias1, user1ApiToken); moveDataset.prettyPrint(); @@ -209,30 +206,30 @@ public void testMoveDatasetNotification() { // verify that a notification was sent to user1 Response getNotifications = UtilIT.getNotifications(user1ApiToken); getNotifications.prettyPrint(); - getNotifications.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.notifications[0].type", equalTo("DATASETMOVED")) - .body("data.notifications[0].displayAsRead", equalTo(false)) - .body("data.notifications[0].subjectText", containsString("has been moved")) - .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + verifyNotification(getNotifications, dataverseAlias1); + // verify that a notification was sent to user2 getNotifications = UtilIT.getNotifications(user2ApiToken); getNotifications.prettyPrint(); - getNotifications.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data.notifications[0].type", equalTo("DATASETMOVED")) - .body("data.notifications[0].displayAsRead", equalTo(false)) - .body("data.notifications[0].subjectText", containsString("has been moved")) - .body("data.notifications[0].messageText", startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + verifyNotification(getNotifications, dataverseAlias1); } - private void clearNotifications(String apiToken) { - Response getNotifications = UtilIT.getNotifications(apiToken); - List notifications = JsonPath.from(getNotifications.body().asString()).getList("data.notifications"); - for (Object obj : notifications) { - Object id = ((Map) obj).get("id"); - UtilIT.deleteNotification(Long.parseLong(id.toString()), apiToken).prettyPrint(); + private void verifyNotification(Response notificationListResponse, String dataverseAlias) { + notificationListResponse.then().assertThat() + .statusCode(OK.getStatusCode()); + boolean found = false; + List> notifications = notificationListResponse.body().jsonPath().getList("data.notifications"); + + for (Map notification : notifications) { + if ("DATASETMOVED".equalsIgnoreCase(notification.get("type"))) { + if (notification.get("messageText") != null && notification.get("messageText").contains(dataverseAlias)) { + found = true; + assertTrue(notification.get("subjectText") != null && notification.get("subjectText").contains("has been moved")); + assertTrue(notification.get("messageText") != null && notification.get("messageText").startsWith(BundleUtil.getStringFromBundle("notification.email.greeting"))); + } + } } + assertTrue(found); } @Test @@ -400,12 +397,12 @@ public void testMoveLinkedDataset() { assertEquals("OK", linksAfterData.getString("status")); assertEquals(0, linksAfterData.getJsonObject("data").getJsonArray("linked-dataverses").size()); } - + @Test public void testMoveDatasetsPerms() { /* - Verify that permissions set on a dataset remain + Verify that permissions set on a dataset remain after that dataaset is moved */ Response createCurator = UtilIT.createRandomUser(); From 84ad0737cc3e9f7fd05a932f76cf0e2cf4a75c3f Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Thu, 23 Oct 2025 17:08:09 -0400 Subject: [PATCH 062/578] remove tabs --- .../iq/dataverse/engine/command/impl/MoveDatasetCommand.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java index 335a28d45b2..a4e43da7dfa 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/MoveDatasetCommand.java @@ -32,8 +32,8 @@ * @author skraffmi */ @RequiredPermissionsMap({ - @RequiredPermissions(dataverseName = "moved", value = {Permission.PublishDataset}) - , @RequiredPermissions(dataverseName = "destination", value = {Permission.AddDataset, Permission.PublishDataset}) + @RequiredPermissions(dataverseName = "moved", value = {Permission.PublishDataset}), + @RequiredPermissions(dataverseName = "destination", value = {Permission.AddDataset, Permission.PublishDataset}) }) public class MoveDatasetCommand extends AbstractVoidCommand { From 4f6d8e813a28945908a10f25403b76b2f79a58e9 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Fri, 24 Oct 2025 10:01:31 -0400 Subject: [PATCH 063/578] Apply suggestion from @pdurbin Co-authored-by: Philip Durbin --- doc/sphinx-guides/source/installation/config.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index a0b4f704d43..f89fe2940ee 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -99,10 +99,10 @@ Still feel like activating this option in your configuration? - Have fun and be Using X-Forwarded-Proto for Signed Urls +++++++++++++++++++++++++++++++++++++++ -If you use an Apache or Nginx proxy, or have a firewall such as Anubis, and they are configured to forward traffic to Dataverse over http -(i.e. your proxy receives user calls over https but forwards locally to Dataverse over http), signed urls, used by external tools and +If you use a proxy such as Apache or Nginx, or have a firewall such as Anubis, and they are configured to forward traffic to Dataverse over HTTP +(i.e. your proxy receives user calls over HTTPS but forwards locally to Dataverse over HTTP), signed URLs, used by external tools and upload apps (such as DVWebloader), are likely to fail unless you configure your proxy to send an X-Forwarded-Proto HTTP Header. -This allows Dataverse to recognize that the communication from the user was over https and that validation of signed urls should assume +This allows Dataverse to recognize that the communication from the user was over HTTPS and that validation of signed URLs should assume they started with https:// (rather than http:// as received from the proxy). .. _PrivacyConsiderations: From 982360587586d4bb71054fa7b99c4f644db47364 Mon Sep 17 00:00:00 2001 From: qqmyers Date: Fri, 24 Oct 2025 10:01:40 -0400 Subject: [PATCH 064/578] Apply suggestion from @pdurbin Co-authored-by: Philip Durbin --- doc/sphinx-guides/source/installation/config.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index f89fe2940ee..8ee3740dea4 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -96,7 +96,7 @@ Still feel like activating this option in your configuration? - Have fun and be .. _signed-urls-forwarded-proto-header: -Using X-Forwarded-Proto for Signed Urls +Using X-Forwarded-Proto for Signed URLs +++++++++++++++++++++++++++++++++++++++ If you use a proxy such as Apache or Nginx, or have a firewall such as Anubis, and they are configured to forward traffic to Dataverse over HTTP From 10184e1bf8934e60f517f2349f9f151005f54b22 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Fri, 24 Oct 2025 10:10:40 -0400 Subject: [PATCH 065/578] addressing comments --- .../iq/dataverse/util/DatasetFieldWalker.java | 19 ++++++++----------- .../iq/dataverse/util/json/JsonPrinter.java | 9 +++++---- .../harvard/iq/dataverse/api/DatasetsIT.java | 8 -------- 3 files changed, 13 insertions(+), 23 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/util/DatasetFieldWalker.java b/src/main/java/edu/harvard/iq/dataverse/util/DatasetFieldWalker.java index 25032860d11..207a12438f9 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/DatasetFieldWalker.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/DatasetFieldWalker.java @@ -5,7 +5,6 @@ import edu.harvard.iq.dataverse.DatasetFieldCompoundValue; import edu.harvard.iq.dataverse.DatasetFieldType; import edu.harvard.iq.dataverse.DatasetFieldValue; -import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import java.util.ArrayList; import java.util.Collections; import java.util.Comparator; @@ -13,7 +12,6 @@ import java.util.Map; import java.util.logging.Logger; -import jakarta.json.Json; import jakarta.json.JsonObject; /** @@ -46,21 +44,20 @@ public interface Listener { */ public static void walk( DatasetField dsf, Listener l, Map cvocMap ) { DatasetFieldWalker joe = new DatasetFieldWalker(l, cvocMap); - SettingsServiceBean nullServiceBean = null; - joe.walk(dsf, nullServiceBean); + joe.walk(dsf, Collections.emptyList()); } /** * Convenience method to walk over a list of fields. Traversal * is done in display order. * @param fields the fields to go over. Does not have to be sorted. - * @param exclude the fields to skip + * @param excludedFieldTypeList the fields to skip * @param l the listener to execute on each field values and structure. */ - public static void walk(List fields, SettingsServiceBean settingsService, Map cvocMap, Listener l) { + public static void walk(List fields, List excludedFieldTypeList, Map cvocMap, Listener l) { DatasetFieldWalker joe = new DatasetFieldWalker(l, cvocMap); for ( DatasetField dsf : sort( fields, DatasetField.DisplayOrder) ) { - joe.walk(dsf, settingsService); + joe.walk(dsf, excludedFieldTypeList); } } @@ -77,7 +74,7 @@ public DatasetFieldWalker(){ this( null, null); } - public void walk(DatasetField fld, SettingsServiceBean settingsService) { + public void walk(DatasetField fld, List excludedFieldTypeList) { l.startField(fld); DatasetFieldType datasetFieldType = fld.getDatasetFieldType(); @@ -89,7 +86,7 @@ public void walk(DatasetField fld, SettingsServiceBean settingsService) { } else if ( datasetFieldType.isPrimitive() ) { for ( DatasetFieldValue pv : sort(fld.getDatasetFieldValues(), DatasetFieldValue.DisplayOrder) ) { - if (settingsService != null && settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false) && DatasetFieldType.FieldType.EMAIL.equals(pv.getDatasetField().getDatasetFieldType().getFieldType())) { + if (excludedFieldTypeList.contains(pv.getDatasetField().getDatasetFieldType().getFieldType())) { continue; } l.primitiveValue(pv); @@ -99,7 +96,7 @@ public void walk(DatasetField fld, SettingsServiceBean settingsService) { for ( DatasetFieldCompoundValue dsfcv : sort( fld.getDatasetFieldCompoundValues(), DatasetFieldCompoundValue.DisplayOrder) ) { l.startCompoundValue(dsfcv); for ( DatasetField dsf : sort(dsfcv.getChildDatasetFields(), DatasetField.DisplayOrder ) ) { - walk(dsf, settingsService); + walk(dsf, excludedFieldTypeList); } l.endCompoundValue(dsfcv); } @@ -107,7 +104,7 @@ public void walk(DatasetField fld, SettingsServiceBean settingsService) { l.addExpandedValuesArray(fld); if(datasetFieldType.isPrimitive() && cvocMap.containsKey(datasetFieldType.getId())) { for ( DatasetFieldValue evv : sort(fld.getDatasetFieldValues(), DatasetFieldValue.DisplayOrder) ) { - if (settingsService != null && settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false) && DatasetFieldType.FieldType.EMAIL.equals(evv.getDatasetField().getDatasetFieldType().getFieldType())) { + if (excludedFieldTypeList.contains(evv.getDatasetField().getDatasetFieldType().getFieldType())) { continue; } l.externalVocabularyValue(evv, cvocMap.get(datasetFieldType.getId())); diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index 842e8ae5be4..4dda90b702c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -693,11 +693,12 @@ public static JsonObjectBuilder json(MetadataBlock block, List fie final JsonArrayBuilder fieldsArray = Json.createArrayBuilder(); Map cvocMap = (datasetFieldService==null) ? new HashMap() :datasetFieldService.getCVocConf(true); - if (ignoreSettingExcludeEmailFromExport) { - DatasetFieldWalker.walk(fields, null, cvocMap, new DatasetFieldsToJson(fieldsArray, anonymizedFieldTypeNamesList)); - } else { - DatasetFieldWalker.walk(fields, settingsService, cvocMap, new DatasetFieldsToJson(fieldsArray, anonymizedFieldTypeNamesList)); + List excludedFieldTypeList = new ArrayList<>(); + // Exclude the Email field or override the exclusion of the Email field type based on the settings ExcludeEmailFromExport and ignoreSettingExcludeEmailFromExport + if (settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false) && !ignoreSettingExcludeEmailFromExport) { + excludedFieldTypeList.add(DatasetFieldType.FieldType.EMAIL); } + DatasetFieldWalker.walk(fields, excludedFieldTypeList, cvocMap, new DatasetFieldsToJson(fieldsArray, anonymizedFieldTypeNamesList)); blockBld.add("fields", fieldsArray); return blockBld; diff --git a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java index 0086fd496c7..a8ad4f55750 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/DatasetsIT.java @@ -76,10 +76,6 @@ public static void setUpClass() { removeIdentifierGenerationStyle.then().assertThat() .statusCode(200); - Response removeExcludeEmail = UtilIT.deleteSetting(SettingsServiceBean.Key.ExcludeEmailFromExport); - removeExcludeEmail.then().assertThat() - .statusCode(200); - Response removeAnonymizedFieldTypeNames = UtilIT.deleteSetting(SettingsServiceBean.Key.AnonymizedFieldTypeNames); removeAnonymizedFieldTypeNames.then().assertThat() .statusCode(200); @@ -110,10 +106,6 @@ public static void afterClass() { removeIdentifierGenerationStyle.then().assertThat() .statusCode(200); - Response removeExcludeEmail = UtilIT.deleteSetting(SettingsServiceBean.Key.ExcludeEmailFromExport); - removeExcludeEmail.then().assertThat() - .statusCode(200); - Response removeAnonymizedFieldTypeNames = UtilIT.deleteSetting(SettingsServiceBean.Key.AnonymizedFieldTypeNames); removeAnonymizedFieldTypeNames.then().assertThat() .statusCode(200); From a52cd0fe963871081c4e60f7e47799eab418109f Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Fri, 24 Oct 2025 10:37:12 -0400 Subject: [PATCH 066/578] fix NPE in test --- .../java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index 4dda90b702c..4ac6585e0f8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -695,7 +695,7 @@ public static JsonObjectBuilder json(MetadataBlock block, List fie Map cvocMap = (datasetFieldService==null) ? new HashMap() :datasetFieldService.getCVocConf(true); List excludedFieldTypeList = new ArrayList<>(); // Exclude the Email field or override the exclusion of the Email field type based on the settings ExcludeEmailFromExport and ignoreSettingExcludeEmailFromExport - if (settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false) && !ignoreSettingExcludeEmailFromExport) { + if ((settingsService != null) && settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false) && !ignoreSettingExcludeEmailFromExport) { excludedFieldTypeList.add(DatasetFieldType.FieldType.EMAIL); } DatasetFieldWalker.walk(fields, excludedFieldTypeList, cvocMap, new DatasetFieldsToJson(fieldsArray, anonymizedFieldTypeNamesList)); From 0e34251b43a6a94bb81ea0bceb58773729449e52 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Fri, 24 Oct 2025 11:33:13 -0400 Subject: [PATCH 067/578] addressing comments --- .../dataverse/GuestbookResponseServiceBean.java | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java index 2393d72dd2a..b37d69cf67b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/GuestbookResponseServiceBean.java @@ -772,22 +772,7 @@ private void setUserDefaultResponses(GuestbookResponse guestbookResponse, Datave } private void setUserDefaultResponses(GuestbookResponse guestbookResponse, DataverseSession session) { - User user = session.getUser(); - - if (user != null) { - guestbookResponse.setEmail(getUserEMail(user)); - guestbookResponse.setName(getUserName(user)); - guestbookResponse.setInstitution(getUserInstitution(user)); - guestbookResponse.setPosition(getUserPosition(user)); - guestbookResponse.setAuthenticatedUser(getAuthenticatedUser(user)); - } else { - guestbookResponse.setEmail(""); - guestbookResponse.setName(""); - guestbookResponse.setInstitution(""); - guestbookResponse.setPosition(""); - guestbookResponse.setAuthenticatedUser(null); - } - guestbookResponse.setSessionId(session.toString()); + setUserDefaultResponses(guestbookResponse, session, session.getUser()); } public GuestbookResponse initDefaultGuestbookResponse(Dataset dataset, DataFile dataFile, DataverseSession session) { From b23ed4ea16bcdbec84997ac09fb5b4a5198b9614 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Fri, 24 Oct 2025 15:13:54 -0400 Subject: [PATCH 068/578] Refactored to use 'enabled' flag to hide provider from JSF --- ...spa-oidc-providers-from-jsf-login-screen.md | 13 +++++++++++++ .../edu/harvard/iq/dataverse/LoginPage.java | 10 +--------- .../authorization/AuthenticationProvider.java | 4 +--- .../AbstractOAuth2AuthenticationProvider.java | 4 ++++ .../oauth2/oidc/OIDCAuthProvider.java | 8 ++------ .../OIDCAuthenticationProviderFactory.java | 8 ++++---- .../iq/dataverse/settings/JvmSettings.java | 1 - src/main/webapp/loginpage.xhtml | 6 +++--- .../edu/harvard/iq/dataverse/api/AdminIT.java | 18 +++++++++++++----- 9 files changed, 41 insertions(+), 31 deletions(-) create mode 100644 doc/release-notes/11606-hide-spa-oidc-providers-from-jsf-login-screen.md diff --git a/doc/release-notes/11606-hide-spa-oidc-providers-from-jsf-login-screen.md b/doc/release-notes/11606-hide-spa-oidc-providers-from-jsf-login-screen.md new file mode 100644 index 00000000000..a5a152a396e --- /dev/null +++ b/doc/release-notes/11606-hide-spa-oidc-providers-from-jsf-login-screen.md @@ -0,0 +1,13 @@ +This release will allow for SPA created OIDC Providers to be hidden from JSF login screens. By setting the ``enabled`` attribute either in the config file ``DATAVERSE_AUTH_OIDC_ENABLED: "0"`` or in the Json of the api call: +POST api/admin/authenticationProviders + +{ +"id": "oidc1", +"factoryAlias": "oidc", +"title": "Open ID Connect SPA", +"subtitle": "SPA OIDC Provider", +"factoryData": "type: oidc | issuer: http://keycloak.mydomain.com:8090/realms/test | clientId: test | clientSecret: 94XHrfNRwXsjqTqApRrwWmhDLDHpIYV8", +```"enabled": false``` +} + +Calling GET api/admin/authenticationProviders will return all providers allowing SPA to display even the ones with enabled = false diff --git a/src/main/java/edu/harvard/iq/dataverse/LoginPage.java b/src/main/java/edu/harvard/iq/dataverse/LoginPage.java index 3580d3b596b..1450ba843ed 100644 --- a/src/main/java/edu/harvard/iq/dataverse/LoginPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/LoginPage.java @@ -135,13 +135,7 @@ public List listCredentialsAuthenticationProv * Retrieve information about all enabled identity providers in a sorted order to be displayed to the user. * @return list of display information for each provider */ - public List listAuthenticationProvidersJSF() { - return listAuthenticationProviders(false); - } public List listAuthenticationProviders() { - return listAuthenticationProviders(true); - } - public List listAuthenticationProviders(boolean ignoreBlocked) { List infos = new LinkedList<>(); List idps = new ArrayList<>(authSvc.getAuthenticationProviders()); @@ -149,9 +143,7 @@ public List listAuthenticationProviders(boole Collections.sort(idps, Comparator.comparing(AuthenticationProvider::getOrder).thenComparing(AuthenticationProvider::getId)); for (AuthenticationProvider idp : idps) { - if (idp != null && (ignoreBlocked || !idp.isJsfBlocked())) { - infos.add(idp.getInfo()); - } + infos.add(idp.getInfo()); } return infos; } diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java index 3a588d15ea8..56e25d18eff 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java @@ -33,9 +33,7 @@ public interface AuthenticationProvider { default boolean isUserInfoUpdateAllowed() { return false; }; default boolean isUserDeletionAllowed() { return false; }; default boolean isOAuthProvider() { return false; }; - default boolean isJsfBlocked() { return false; }; - - + default boolean isEnabled() { return true; }; /** * Some providers (e.g organizational ones) provide verified email addresses. diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java index a6b7c1b9d49..45efa68e9f7 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java @@ -93,6 +93,7 @@ public String toString() { protected String clientSecret; protected String baseUserEndpoint; protected String redirectUrl; + protected boolean enabled = true; /** * List of scopes to be requested for authorization at identity provider. @@ -272,6 +273,9 @@ public String getSubTitle() { public String getSpacedScope() { return String.join(" ", getScope()); } + public void setEnabled(boolean enabled) { this.enabled = enabled; } + public boolean isEnabled() { return enabled; } + @Override public int hashCode() { int hash = 7; diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java index d9c69920cb3..58c6edda159 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java @@ -67,8 +67,7 @@ public class OIDCAuthProvider extends AbstractOAuth2AuthenticationProvider { final OIDCProviderMetadata idpMetadata; final boolean pkceEnabled; final CodeChallengeMethod pkceMethod; - final boolean jsfBlocked; - + /** * Using PKCE, we create and send a special {@link CodeVerifier}. This contains a secret * we need again when verifying the response by the provider, thus the cache. @@ -81,7 +80,7 @@ public class OIDCAuthProvider extends AbstractOAuth2AuthenticationProvider { .build(); public OIDCAuthProvider(String aClientId, String aClientSecret, String issuerEndpointURL, - boolean pkceEnabled, String pkceMethod, boolean jsfBlocked) throws AuthorizationSetupException { + boolean pkceEnabled, String pkceMethod) throws AuthorizationSetupException { this.clientSecret = aClientSecret; // needed for state creation this.clientAuth = new ClientSecretBasic(new ClientID(aClientId), new Secret(aClientSecret)); this.issuer = new Issuer(issuerEndpointURL); @@ -90,10 +89,7 @@ public OIDCAuthProvider(String aClientId, String aClientSecret, String issuerEnd this.pkceEnabled = pkceEnabled; this.pkceMethod = CodeChallengeMethod.parse(pkceMethod); - this.jsfBlocked = jsfBlocked; } - - public boolean isJsfBlocked() { return jsfBlocked; } /** * Setup metadata from OIDC provider during creation of the provider representation diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java index 7a2f6acc41c..70223638663 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthenticationProviderFactory.java @@ -43,13 +43,13 @@ public AuthenticationProvider buildProvider( AuthenticationProviderRow aRow ) th factoryData.get("clientSecret"), factoryData.get("issuer"), Boolean.parseBoolean(factoryData.getOrDefault("pkceEnabled", "false")), - factoryData.getOrDefault("pkceMethod", "S256"), - Boolean.parseBoolean(factoryData.getOrDefault("jsfBlocked", "false")) + factoryData.getOrDefault("pkceMethod", "S256") ); oidc.setId(aRow.getId()); oidc.setTitle(aRow.getTitle()); oidc.setSubTitle(aRow.getSubtitle()); + oidc.setEnabled(aRow.isEnabled()); return oidc; } @@ -65,13 +65,13 @@ public static AuthenticationProvider buildFromSettings() throws AuthorizationSet JvmSettings.OIDC_CLIENT_SECRET.lookup(), JvmSettings.OIDC_AUTH_SERVER_URL.lookup(), JvmSettings.OIDC_PKCE_ENABLED.lookupOptional(Boolean.class).orElse(false), - JvmSettings.OIDC_PKCE_METHOD.lookupOptional().orElse("S256"), - JvmSettings.OIDC_JSFBLOCKED.lookupOptional(Boolean.class).orElse(false) + JvmSettings.OIDC_PKCE_METHOD.lookupOptional().orElse("S256") ); oidc.setId("oidc-mpconfig"); oidc.setTitle(JvmSettings.OIDC_TITLE.lookupOptional().orElse("OpenID Connect")); oidc.setSubTitle(JvmSettings.OIDC_SUBTITLE.lookupOptional().orElse("OpenID Connect")); + oidc.setEnabled(JvmSettings.OIDC_ENABLED.lookupOptional(Boolean.class).orElse(true)); return oidc; } diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java index 77c12ec3bd4..87123801a3e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java @@ -252,7 +252,6 @@ public enum JvmSettings { OIDC_AUTH_SERVER_URL(SCOPE_OIDC, "auth-server-url"), OIDC_CLIENT_ID(SCOPE_OIDC, "client-id"), OIDC_CLIENT_SECRET(SCOPE_OIDC, "client-secret"), - OIDC_JSFBLOCKED(SCOPE_OIDC, "jsfBlocked"), SCOPE_OIDC_PKCE(SCOPE_OIDC, "pkce"), OIDC_PKCE_ENABLED(SCOPE_OIDC_PKCE, "enabled"), OIDC_PKCE_METHOD(SCOPE_OIDC_PKCE, "method"), diff --git a/src/main/webapp/loginpage.xhtml b/src/main/webapp/loginpage.xhtml index 498f6e53ff3..ffb2ce0f935 100644 --- a/src/main/webapp/loginpage.xhtml +++ b/src/main/webapp/loginpage.xhtml @@ -23,7 +23,7 @@ - +
          @@ -214,10 +214,10 @@
          -
          +

          #{bundle['auth.providers.title']}

          - + diff --git a/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java b/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java index 45765b88ad5..012f56c2760 100644 --- a/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java +++ b/src/test/java/edu/harvard/iq/dataverse/api/AdminIT.java @@ -978,14 +978,14 @@ public void testAddAuthProviders() { Response getAuthProviders = UtilIT.getAuthProviders(superuserApiToken); getAuthProviders.prettyPrint(); - String factoryData = String.format("type: oidc | issuer: http://keycloak.mydomain.com:8090/realms/test | clientId: %s | clientSecret: %s | jsfBlocked: true", clientId, clientSecret); + String factoryData = String.format("type: oidc | issuer: http://keycloak.mydomain.com:8090/realms/test | clientId: %s | clientSecret: %s", clientId, clientSecret); JsonObject jsonObject = Json.createObjectBuilder() .add("id", "oidc1") .add("factoryAlias", "oidc") .add("title", "Open ID Connect SPA") .add("subtitle", "SPA OIDC Provider") .add("factoryData", factoryData) - .add("enabled", true) + .add("enabled", false) .build(); Response addAuthProviders = UtilIT.addAuthProviders(superuserApiToken, jsonObject); addAuthProviders.prettyPrint(); @@ -995,9 +995,17 @@ public void testAddAuthProviders() { getAuthProviders = UtilIT.getAuthProviders(superuserApiToken); getAuthProviders.prettyPrint(); getAuthProviders.then().assertThat() - .statusCode(OK.getStatusCode()) - .body("data[1].id", equalTo("oidc1")) - .body("data[1].factoryData", containsString("jsfBlocked: true")); + .statusCode(OK.getStatusCode()); + boolean found = false; + List> providers = getAuthProviders.body().jsonPath().getList("data"); + for (Map provider : providers) { + if ("oidc1".equalsIgnoreCase((String) provider.get("id"))) { + found = true; + assertTrue(provider.get("title") != null && provider.get("title").equals("Open ID Connect SPA")); + assertTrue(provider.get("enabled") != null && !(Boolean) provider.get("enabled")); + } + } + assertTrue(found); } } From f4a4ed504e6f3b8c3f8a5c5dc1a6df623ec6f9db Mon Sep 17 00:00:00 2001 From: qqmyers Date: Fri, 24 Oct 2025 16:50:42 -0400 Subject: [PATCH 069/578] add space to fix link --- doc/sphinx-guides/source/developers/workflows.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/developers/workflows.rst b/doc/sphinx-guides/source/developers/workflows.rst index a7f6c8a044c..8248144d4d8 100644 --- a/doc/sphinx-guides/source/developers/workflows.rst +++ b/doc/sphinx-guides/source/developers/workflows.rst @@ -205,7 +205,7 @@ Note - the example step includes two settings required for any archiver, three ( ldnannounce +++++++++++ -A step that sends a `Linked Data Notification (LDN)`_ message to a specific LDN Inbox announcing a relationship between an newly published/available dataset with a relationship to an external resource (e.g. one managed by the recipient). +A step that sends a `Linked Data Notification (LDN) `_ message to a specific LDN Inbox announcing a relationship between an newly published/available dataset with a relationship to an external resource (e.g. one managed by the recipient). The two parameters are * ``:LDNAnnounceRequiredFields`` - a list of metadata fields that must exist to trigger the message. Currently, the message also includes the values for these fields but future versions may only send the dataset's persistent identifier (making the receiver responsible for making a call-back to get any metadata). From 206e0afc3950e7e357c84b861edaf34ac68e4d3c Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Mon, 27 Oct 2025 10:14:44 -0400 Subject: [PATCH 070/578] fix checkstyle --- .../oauth2/AbstractOAuth2AuthenticationProvider.java | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java index 45efa68e9f7..d72af868fd6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java @@ -273,8 +273,13 @@ public String getSubTitle() { public String getSpacedScope() { return String.join(" ", getScope()); } - public void setEnabled(boolean enabled) { this.enabled = enabled; } - public boolean isEnabled() { return enabled; } + public void setEnabled(boolean enabled) { + this.enabled = enabled; + } + + public boolean isEnabled() { + return enabled; + } @Override public int hashCode() { From 64e1d0257874d56bfb83d1b3579d99a3101d34ee Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Mon, 27 Oct 2025 11:05:52 -0400 Subject: [PATCH 071/578] fix checkstyle --- .../iq/dataverse/authorization/AuthenticationProvider.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java index 56e25d18eff..79ecbe9c0db 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/AuthenticationProvider.java @@ -33,7 +33,9 @@ public interface AuthenticationProvider { default boolean isUserInfoUpdateAllowed() { return false; }; default boolean isUserDeletionAllowed() { return false; }; default boolean isOAuthProvider() { return false; }; - default boolean isEnabled() { return true; }; + default boolean isEnabled() { + return true; + }; /** * Some providers (e.g organizational ones) provide verified email addresses. From 787f9a107529fb324d499df82abd3fb819c69e77 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Mon, 27 Oct 2025 14:13:34 -0400 Subject: [PATCH 072/578] revert LoginPage.java --- src/main/java/edu/harvard/iq/dataverse/LoginPage.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/LoginPage.java b/src/main/java/edu/harvard/iq/dataverse/LoginPage.java index 1450ba843ed..ecc249cee87 100644 --- a/src/main/java/edu/harvard/iq/dataverse/LoginPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/LoginPage.java @@ -143,7 +143,9 @@ public List listAuthenticationProviders() { Collections.sort(idps, Comparator.comparing(AuthenticationProvider::getOrder).thenComparing(AuthenticationProvider::getId)); for (AuthenticationProvider idp : idps) { - infos.add(idp.getInfo()); + if (idp != null) { + infos.add(idp.getInfo()); + } } return infos; } From 65be513c4535c3cc23fa6463eef39fa169426453 Mon Sep 17 00:00:00 2001 From: Steven Winship <39765413+stevenwinship@users.noreply.github.com> Date: Mon, 27 Oct 2025 14:24:47 -0400 Subject: [PATCH 073/578] revert OIDCAuthProvider.java --- .../providers/oauth2/oidc/OIDCAuthProvider.java | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java index 58c6edda159..3e62598ee79 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/oidc/OIDCAuthProvider.java @@ -67,7 +67,7 @@ public class OIDCAuthProvider extends AbstractOAuth2AuthenticationProvider { final OIDCProviderMetadata idpMetadata; final boolean pkceEnabled; final CodeChallengeMethod pkceMethod; - + /** * Using PKCE, we create and send a special {@link CodeVerifier}. This contains a secret * we need again when verifying the response by the provider, thus the cache. @@ -81,7 +81,7 @@ public class OIDCAuthProvider extends AbstractOAuth2AuthenticationProvider { public OIDCAuthProvider(String aClientId, String aClientSecret, String issuerEndpointURL, boolean pkceEnabled, String pkceMethod) throws AuthorizationSetupException { - this.clientSecret = aClientSecret; // needed for state creation + this.clientSecret = aClientSecret; // nedded for state creation this.clientAuth = new ClientSecretBasic(new ClientID(aClientId), new Secret(aClientSecret)); this.issuer = new Issuer(issuerEndpointURL); @@ -90,7 +90,8 @@ public OIDCAuthProvider(String aClientId, String aClientSecret, String issuerEnd this.pkceEnabled = pkceEnabled; this.pkceMethod = CodeChallengeMethod.parse(pkceMethod); } - + + /** * Setup metadata from OIDC provider during creation of the provider representation * @return The OIDC provider metadata, if successfull From 0eeac9793448544eb8fe2f1e0dbf48675ff3f089 Mon Sep 17 00:00:00 2001 From: Stephen Kraffmiller Date: Mon, 27 Oct 2025 16:51:43 -0400 Subject: [PATCH 074/578] #11918 prelim check in --- .../iq/dataverse/TemplateServiceBean.java | 3 +- .../harvard/iq/dataverse/api/Dataverses.java | 36 ++++++++++++++++++- .../command/impl/CreateTemplateCommand.java | 10 +++--- .../iq/dataverse/api/DataversesIT.java | 23 ++++++++++-- .../edu/harvard/iq/dataverse/api/UtilIT.java | 14 ++++++++ 5 files changed, 78 insertions(+), 8 deletions(-) diff --git a/src/main/java/edu/harvard/iq/dataverse/TemplateServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/TemplateServiceBean.java index 46382fc2588..7093f610979 100644 --- a/src/main/java/edu/harvard/iq/dataverse/TemplateServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/TemplateServiceBean.java @@ -30,7 +30,8 @@ public Template find(Object pk) { } public Template save(Template template) { - return em.merge(template); + Template merged = em.merge(template); + return merged; } public List