You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+18-12Lines changed: 18 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ You will need Python 3.9 or higher installed, as well as pip. You can run `pip i
26
26
27
27
We will create a simple example pipeline from a [PokeAPI spec](https://pokeapi.co/) in our repo. You can point to any other OpenAPI Spec instead if you prefer.
# 2. You can now pick both of the endpoints from the popup.
37
37
38
-
# 3. After selecting your Pokemon endpoints and hitting Enter, your pipeline will be rendered.
38
+
# 3. After selecting your Pokemon endpoints and hitting Enter,
39
+
# your pipeline will be rendered.
39
40
40
-
# 4. If you have any kind of authentication on your pipeline (this example does not), open the `.dlt/secrets.toml` and provide the credentials. You can find further settings in the `.dlt/config.toml`.
41
+
# 4. If you have any kind of authentication on your pipeline (this example does not),
42
+
# open the `.dlt/secrets.toml` and provide the credentials. You can find further
43
+
# settings in the `.dlt/config.toml`.
41
44
42
45
# 5. Go to the created pipeline folder and run your pipeline.
43
46
$ cd pokemon-pipeline
@@ -46,22 +49,25 @@ $ PROGRESS=enlighten python pipeline.py # we use enlighten for a nice progress b
46
49
# 6. Print the pipeline info to the console to see what got loaded.
47
50
$ dlt pipeline pokemon_pipeline info
48
51
49
-
# 7. You can now also install Streamlit to see a preview of the data; you should have loaded 40 Pokemons and their details.
52
+
# 7. You can now also install Streamlit to see a preview of the data; you should
53
+
# have loaded 40 Pokemons and their details.
50
54
$ pip install pandas streamlit
51
55
$ dlt pipeline pokemon_pipeline show
52
56
53
-
# 8. You can go to our docs at https://dlthub.com/docs to learn how to modify the generated pipeline to load to many destinations, place schema contracts on your pipeline, and many other things.
57
+
# 8. You can go to our docs at https://dlthub.com/docs to learn how to modify
58
+
# the generated pipeline to load to many destinations, place schema contracts
59
+
# on your pipeline, and many other things.
54
60
55
-
# NOTE: We used the `--global-limit 2` CLI flag to limit the requests to the PokeAPI forthis example. This way, the Pokemon collection endpoint only gets queried twice, resultingin 2 x 20 Pokemon
56
-
57
-
details being rendered.
61
+
# NOTE: We used the `--global-limit 2` CLI flag to limit the requests to the PokeAPI
62
+
# for this example. This way, the Pokemon collection endpoint only gets queried
63
+
# twice, resulting in 2 x 20 Pokemon details being rendered.
58
64
```
59
65
60
66
## What will be created?
61
67
62
68
When you run the `init` command above, the following files will be generated:
63
69
64
-
```
70
+
```text
65
71
pokemon_pipeline/
66
72
├── .dlt/
67
73
│ ├── config.toml # dlt config, learn more at dlthub.com/docs
@@ -84,7 +90,7 @@ This file contains the configuration dictionary for the [dlt rest_api](https://d
84
90
85
91
The generated dictionary will look something like this:
86
92
87
-
```python
93
+
```py
88
94
{
89
95
"client": {
90
96
"base_url": base_url,
@@ -134,7 +140,7 @@ The generated dictionary will look something like this:
0 commit comments