Add CREATE JOB DDL by ryannedolan · Pull Request #215 · linkedin/Hoptimator

ryannedolan · 2026-04-28T03:18:39Z

Summary

Added create job ddl.

Details

Adds create [or replace] [flink] [streaming | batch] job as 'insert into ... ' [with (... options ...)] statement. This enables automation to bypass Hoptimator's pipeline planner and create Flink jobs directly. This is particularly useful if the target Flink environment includes a pre-configured catalog, since in that case jobs can be simple one-liners.

Testing Done

> create flink streaming job "my-etl-job" as 'INSERT INTO sink SELECT * FROM source' WITH (checkpointing.interval '10000');
apiVersion: hoptimator.linkedin.com/v1alpha1

produces:

kind: SqlJob
metadata:
  name: my-etl-job
spec:
  configs:
    checkpointing.interval: "10000"
  dialect: Flink
  executionMode: Streaming
  sql:
  - INSERT INTO sink SELECT * FROM source

Introduces CREATE [OR REPLACE] DATABASE <name> [WITH (...)] syntax that deploys a Database CRD to Kubernetes via the Deployer pattern. Includes parser, executor, shared processCreateDatabase util, K8sDatabaseDeployer, and quidem test with !specify support. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Introduces CREATE JOB syntax that deploys a SqlJob CRD to Kubernetes via the Deployer pattern. Supports optional FLINK dialect and STREAMING/BATCH execution mode keywords, SQL body as a string literal, and WITH options for configs. Since JOB/FLINK/STREAMING/BATCH are not parser tokens, the grammar uses IDENTIFIER matching with text checks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replaces the IDENTIFIER-matching hack with properly generated parser tokens. Regenerated all three parser files (Impl, Constants, TokenManager) from Calcite 1.41.0 codegen using the updated config.fmpp and parserImpls.ftl templates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

SQL parser uppercases unquoted identifiers, so WITH (url '...') produces key "URL". Use a case-insensitive TreeMap in K8sDatabaseDeployer and fix expected config key casing in the CREATE JOB quidem test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

SnakeYAML serializes Java enums by name (MYSQL) not value (MySQL). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ryannedolan · 2026-04-30T21:53:46Z

@copilot please resolve merge conflicts

ryannedolan · 2026-04-30T21:55:21Z

@copilot please add unit tests similar to #214

…CREATE JOB and CREATE DATABASE DDL additions Co-authored-by: ryannedolan <1387539+ryannedolan@users.noreply.github.com>

Copilot · 2026-04-30T22:04:30Z

@copilot please resolve merge conflicts

Resolved in commit 6e9a8d6. Merged origin/main (which had 3 new commits including the CREATE DATABASE DDL feature) into the branch and resolved conflicts in 7 files: config.fmpp, parserImpls.ftl, HoptimatorDdlExecutor.java, HoptimatorDdlUtils.java, HoptimatorDdlParserImpl.java, K8sDeployerProvider.java, and TestSqlScripts.java. Both CREATE JOB and CREATE DATABASE features are preserved.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

checkstyle.org
- Triggering command: /opt/hostedtoolcache/CodeQL/2.25.1/x64/codeql/tools/linux64/java/bin/java /opt/hostedtoolcache/CodeQL/2.25.1/x64/codeql/tools/linux64/java/bin/java -jar /opt/hostedtoolcache/CodeQL/2.25.1/x64/codeql/xml/tools/xml-extractor.jar --fileList=/tmp/codeql-scratch-c2bb8f7fb6eb8b71/dbs/java/working/files-to-index5026978359159947465.list --sourceArchiveDir=/tmp/codeql-scratch-c2bb8f7fb6eb8b71/dbs/java/src --outputDir=/tmp/codeql-scratch-c2bb8f7fb6eb8b71/dbs/java/trap/java m/linkedin/hoptimator/jdbc/HoptimatorDdlExecutor.java tnet/tools/bash (dns block)
linkedin.jfrog.io
- Triggering command: /usr/lib/jvm/temurin-17-jdk-amd64/bin/java /usr/lib/jvm/temurin-17-jdk-amd64/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.xml/javax.xml.namespace=ALL-UNNAMED -XX:MaxMetaspaceSize=384m -XX:+HeapDumpOnOutOfMemoryError -Xms256m -Xmx512m -Dfile.encoding=UTF-8 -Duser.country -Duser.language=en (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

…teJob, K8sDeployerProvider)" Agent-Logs-Url: https://github.com/linkedin/Hoptimator/sessions/4fdc6aed-fd8d-4af3-bbf0-a706fba9773b Co-authored-by: ryannedolan <1387539+ryannedolan@users.noreply.github.com>

…anatory comments Agent-Logs-Url: https://github.com/linkedin/Hoptimator/sessions/4fdc6aed-fd8d-4af3-bbf0-a706fba9773b Co-authored-by: ryannedolan <1387539+ryannedolan@users.noreply.github.com>

Copilot · 2026-04-30T22:22:18Z

@copilot please add unit tests similar to #214

Added in commit 45e9ad1 (minor comment cleanup in bdf0589):

K8sSqlJobDeployerTest (11 tests) — mirrors K8sDatabaseDeployerTest: covers specify() with various dialects/execution modes, options/configs, multiple SQL statements, invalid enum values, name canonicalization, API version/kind, and create().
K8sDeployerProviderTest — added deployersForSqlJobDeployableReturnsSqlJobDeployer().
HoptimatorDdlUtilsTest (5 tests) — processCreateJob and specifyFromSql tests: specify-mode returns empty specs with no deployers, view path contains job name, compound name throws, empty SQL body throws, and specifyFromSql routing.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

checkstyle.org
- Triggering command: /opt/hostedtoolcache/CodeQL/2.25.1/x64/codeql/tools/linux64/java/bin/java /opt/hostedtoolcache/CodeQL/2.25.1/x64/codeql/tools/linux64/java/bin/java -jar /opt/hostedtoolcache/CodeQL/2.25.1/x64/codeql/xml/tools/xml-extractor.jar --fileList=/tmp/codeql-scratch-c2bb8f7fb6eb8b71/dbs/java/working/files-to-index4382818968982770359.list --sourceArchiveDir=/tmp/codeql-scratch-c2bb8f7fb6eb8b71/dbs/java/src --outputDir=/tmp/codeql-scratch-c2bb8f7fb6eb8b71/dbs/java/trap/java --global nfig/composer/vendor/bin/bash gpg.program (dns block)
linkedin.jfrog.io
- Triggering command: /usr/lib/jvm/temurin-17-jdk-amd64/bin/java /usr/lib/jvm/temurin-17-jdk-amd64/bin/java --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.xml/javax.xml.namespace=ALL-UNNAMED -XX:MaxMetaspaceSize=384m -XX:+HeapDumpOnOutOfMemoryError -Xms256m -Xmx512m -Dfile.encoding=UTF-8 -Duser.country -Duser.language=en (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

github-actions · 2026-05-01T00:02:26Z

Code Coverage

Overall Project	84.23% `-0.22%`	🟢
Files changed	80.19%	🟢

File	Coverage
K8sApiEndpoints.java	100%	🟢
K8sDeployer.java	100%	🟢
K8sSqlJobDeployer.java	100%	🟢
K8sDeployerProvider.java	100%	🟢
HoptimatorDdlExecutor.java	89.78% `-3.16%`	❌
SqlJobDeployable.java	89.19% `-10.81%`	🟢
HoptimatorDdlUtils.java	84.93% `-1.59%`	🟢

SnakeYAML's RepresentEnum uses Enum.name() — the Java constant name — rather than toString(), which the generated model enums override to return the CRD-defined string value. Replace Yaml.dump() in K8sDeployer.specify() with a custom SnakeYAML representer that uses toString() for enums, preserving the kubernetes client's null-skipping and alphabetical-sort behaviors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The tests were asserting FLINK/BATCH (Enum.name()) and had comments explaining this was broken behavior. Update them to assert Flink/Batch (Enum.toString() / CRD value) now that dumpYaml() fixes enum serialization. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ryannedolan and others added 5 commits April 27, 2026 21:02

Fix expected dialect enum serialization in quidem test

3e0f3d9

SnakeYAML serializes Java enums by name (MYSQL) not value (MySQL). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Base automatically changed from rdolan/create-database to main April 30, 2026 21:48

Copilot started work on behalf of ryannedolan April 30, 2026 21:53 View session

Merge origin/main into rdolan/create-job - resolve conflicts between …

6e9a8d6

…CREATE JOB and CREATE DATABASE DDL additions Co-authored-by: ryannedolan <1387539+ryannedolan@users.noreply.github.com>

Copilot finished work on behalf of ryannedolan April 30, 2026 22:05

Copilot started work on behalf of ryannedolan April 30, 2026 22:05 View session

Copilot AI and others added 2 commits April 30, 2026 22:17

Add unit tests for CREATE JOB DDL (K8sSqlJobDeployerTest, processCrea…

45e9ad1

…teJob, K8sDeployerProvider)" Agent-Logs-Url: https://github.com/linkedin/Hoptimator/sessions/4fdc6aed-fd8d-4af3-bbf0-a706fba9773b Co-authored-by: ryannedolan <1387539+ryannedolan@users.noreply.github.com>

Address code review: fix canonicalizeName assertion message, add expl…

bdf0589

…anatory comments Agent-Logs-Url: https://github.com/linkedin/Hoptimator/sessions/4fdc6aed-fd8d-4af3-bbf0-a706fba9773b Co-authored-by: ryannedolan <1387539+ryannedolan@users.noreply.github.com>

Copilot finished work on behalf of ryannedolan April 30, 2026 22:23

ryannedolan and others added 5 commits April 30, 2026 22:12

Fix unescaped identifier in test

d02a019

Fix quidem test yaml escaping

a4afdec

Merge branch 'main' into rdolan/create-job

d439a97

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CREATE JOB DDL#215

Add CREATE JOB DDL#215
ryannedolan wants to merge 13 commits intomainfrom
rdolan/create-job

ryannedolan commented Apr 28, 2026

Uh oh!

ryannedolan commented Apr 30, 2026

Uh oh!

ryannedolan commented Apr 30, 2026

Uh oh!

Copilot AI commented Apr 30, 2026 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI commented Apr 30, 2026 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

github-actions Bot commented May 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ryannedolan commented Apr 28, 2026

Summary

Details

Testing Done

Uh oh!

ryannedolan commented Apr 30, 2026

Uh oh!

ryannedolan commented Apr 30, 2026

Uh oh!

Copilot AI commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

github-actions Bot commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Apr 30, 2026 •

edited

Loading

Copilot AI commented Apr 30, 2026 •

edited

Loading

github-actions Bot commented May 1, 2026 •

edited

Loading