Skip to content

Commit 3e1d04c

Browse files
dmikurubedmikurube
authored
Merge pull request #2 from embulk/build-embulk-standards-out-of-embulk-repo
Build embulk-standards plugins out of the embulk repository
2 parents 599a520 + a13a9bb commit 3e1d04c

File tree

24 files changed

+301
-792
lines changed

24 files changed

+301
-792
lines changed

‎.github/workflows/check.yml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: Check
2+
on: [ pull_request, push ]
3+
jobs:
4+
check:
5+
runs-on: ${{ matrix.os }}
6+
strategy:
7+
fail-fast: false
8+
matrix:
9+
os:
10+
- ubuntu-latest
11+
- macOS-latest
12+
# - windows-latest
13+
gradle_task:
14+
- ":embulk-decoder-bzip2:check"
15+
- ":embulk-decoder-gzip:check"
16+
- ":embulk-encoder-bzip2:check"
17+
- ":embulk-encoder-gzip:check"
18+
- ":embulk-filter-remove_columns:check"
19+
- ":embulk-filter-rename:check"
20+
- ":embulk-formatter-csv:check"
21+
- ":embulk-guess-bzip2:check"
22+
- ":embulk-guess-csv:check"
23+
- ":embulk-guess-csv_all_strings:check"
24+
- ":embulk-guess-gzip:check"
25+
- ":embulk-guess-json:check"
26+
- ":embulk-input-config:check"
27+
- ":embulk-input-file:check"
28+
- ":embulk-output-file:check"
29+
- ":embulk-output-null:check"
30+
- ":embulk-output-stdout:check"
31+
- ":embulk-parser-csv:check"
32+
- ":embulk-parser-json:check"
33+
steps:
34+
- uses: actions/checkout@v2
35+
- name: Set up OpenJDK 8
36+
uses: actions/setup-java@v2
37+
with:
38+
java-version: 8
39+
distribution: "zulu"
40+
- name: Build and test
41+
run: ./gradlew ${{ matrix.gradle_task }}

‎.gitignore‎

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,3 @@
1-
target/
2-
build/
3-
pkg/
4-
*.iml
51
*~
6-
._*
7-
.idea
8-
tmp/
9-
vendor/
10-
/classpath/
11-
/.bundle
12-
.yardoc
13-
/embulk-*.jar
2+
build/
143
/.gradle
15-
/coverage

‎README.md‎

Lines changed: 4 additions & 128 deletions
Original file line numberDiff line numberDiff line change
@@ -1,132 +1,8 @@
1-
# What's Embulk?
1+
# embulk-standards: Embulk's standard plugins
22

3-
Embulk is a parallel bulk data loader that **helps data transfer between various storages, databases, NoSQL and cloud services**.
3+
These are Embulk's "standard" plugins which are embedded in Embulk's executable binary distributions.
44

5-
**Embulk supports plugins** to add functions. You can [share the plugins](https://plugins.embulk.org/) to keep your custom scripts readable, maintainable, and reusable.
6-
7-
[![Embulk](https://gist.githubusercontent.com/frsyuki/f322a77ee2766a508ba9/raw/e8539b6b4fda1b3357e8c79d3966aa8148dbdbd3/embulk-overview.png)](http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed/12)
8-
[Embulk, an open-source plugin-based parallel bulk data loader](http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed) at Slideshare
9-
10-
# Document
11-
12-
Embulk documents: https://www.embulk.org/
13-
14-
### Using plugins
15-
16-
You can use plugins to load data from/to various systems and file formats. Here is the list of publicly released plugins: [list of plugins by category](https://plugins.embulk.org/).
17-
18-
An example is [embulk-output-command](https://github.com/embulk/embulk-output-command) plugin. It executes an external command to output the records.
19-
20-
To install plugins, you can use `embulk gem install <name>` command:
21-
22-
```
23-
embulk gem install embulk-output-command
24-
embulk gem list
25-
```
26-
27-
Embulk bundles some built-in plugins such as `embulk-encoder-gzip` or `embulk-formatter-csv`. You can use those plugins with following configuration file:
28-
29-
```yaml
30-
in:
31-
type: file
32-
path_prefix: "./try1/csv/sample_"
33-
...
34-
out:
35-
type: command
36-
command: "cat - > task.$INDEX.$SEQID.csv.gz"
37-
encoders:
38-
- {type: gzip}
39-
formatter:
40-
type: csv
41-
```
42-
43-
### Resuming a failed transaction
44-
45-
Embulk supports resuming failed transactions.
46-
To enable resuming, you need to start transaction with `-r PATH` option:
47-
48-
```
49-
embulk run config.yml -r resume-state.yml
50-
```
51-
52-
If the transaction fails, embulk stores state some states to the yaml file. You can retry the transaction using exactly same command:
53-
54-
```
55-
embulk run config.yml -r resume-state.yml
56-
```
57-
58-
If you give up on resuming the transaction, you can use `embulk cleanup` subcommand to delete intermediate data:
59-
60-
```
61-
embulk cleanup config.yml -r resume-state.yml
62-
```
63-
64-
### Using plugin bundle
65-
66-
`embulk mkbundle` subcommand creates a isolated bundle of plugins. You can install plugins (gems) to the bundle directory instead of ~/.embulk directory. This makes it easy to manage versions of plugins.
67-
To use the bundle, add `-b <bundle_dir>` option to `guess`, `preview`, or `run` subcommand. `embulk mkbundle` also generates some example plugins to \<bundle_dir>/embulk/\*.rb directory.
68-
69-
See the generated \<bundle_dir>/Gemfile file how to plugin bundles work.
70-
71-
```
72-
embulk mkbundle ./embulk_bundle # please edit ./embulk_bundle/Gemfile to add plugins. Detailed usage is written in the Gemfile
73-
embulk guess -b ./embulk_bundle ...
74-
embulk run -b ./embulk_bundle ...
75-
```
76-
77-
## Use cases
78-
79-
* [Scheduled bulk data loading to Elasticsearch + Kibana 5 from CSV files](https://www.embulk.org/recipes/scheduled-csv-load-to-elasticsearch-kibana5.html)
80-
81-
For further details, visit [Embulk documentation](https://www.embulk.org/).
82-
83-
## Upgrading to the latest version
84-
85-
Following command updates embulk itself to the specific released version.
86-
87-
```sh
88-
embulk selfupdate x.y.z
89-
```
90-
91-
## Embulk Development
92-
93-
### Build
94-
95-
```
96-
./gradlew cli # creates pkg/embulk-VERSION.jar
97-
```
98-
99-
You can see JaCoCo's test coverage report at `${project}/build/reports/tests/index.html`
100-
You can see Findbug's report at `${project}/build/reports/findbug/main.html` # FIXME coverage information is not included somehow
101-
102-
You can use `classpath` task to use `bundle exec ./bin/embulk` for development:
103-
104-
```
105-
./gradlew -t classpath # -x test: skip test
106-
./bin/embulk
107-
```
108-
109-
To deploy artifacts to your local maven repository at ~/.m2/repository/:
110-
111-
```
112-
./gradlew install
113-
```
114-
115-
To compile the source code of embulk-core project only:
116-
117-
```
118-
./gradlew :embulk-core:compileJava
119-
```
120-
121-
Task `dependencies` shows dependency tree of embulk-core project:
122-
123-
```
124-
./gradlew :embulk-core:dependencies
125-
```
126-
127-
### Update JRuby
128-
129-
Modify `jrubyVersion` in `build.gradle` to update JRuby of Embulk.
5+
Their source code had been managed in the same [main repository of Embulk](https://github.com/embulk/embulk) until [`v0.10.33`](https://github.com/embulk/embulk/tree/v0.10.33). They have been split from the main repository since `v0.10.34`.
1306

1317
### Release
1328

@@ -151,7 +27,7 @@ signing.secretKeyRingFile=(the absolute path to the secret key ring file contain
15127

15228
#### Release
15329

154-
Modify `version` in `build.gradle` at a detached commit to bump Embulk version up.
30+
Modify `version` in `build.gradle` at a detached commit to bump up the versions of Embulk standard plugins.
15531

15632
```
15733
git checkout --detach master

0 commit comments

Comments
 (0)