You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/collection.jl
+83-26Lines changed: 83 additions & 26 deletions
Original file line number
Diff line number
Diff line change
@@ -47,8 +47,12 @@ end
47
47
48
48
Writes `data`/`ccd` in FITS format at `file_path`.
49
49
50
-
`FITSIO` takes over memory write in by `cfitsio`, which writes in row-major form, whereas when Julia gives that memory, it is assumed as column major.
51
-
Therefore all data written by [`FITSIO.write`](http://juliaastro.github.io/FITSIO.jl/latest/api.html#Base.write-Tuple{FITS,Dict{String,V}%20where%20V}) is transposed. This function allows the user to write the data in a consistent way to FITS file by transposing before writing.
50
+
`FITSIO` takes over memory write in by `cfitsio`, which writes in row-major
51
+
form, whereas when Julia gives that memory, it is assumed as column major.
Walk through `dir` collecting FITS files, scanning their headers, and culminating into a `DataFrame` that can be used with the generators for iterating over many files and processing them. If `recursive` is false, no subdirectories will be walked through.
78
+
Walk through `dir` collecting FITS files, scanning their headers, and
79
+
culminating into a `DataFrame` that can be used with the generators for
80
+
iterating over many files and processing them. If `recursive` is false, no
81
+
subdirectories will be walked through.
75
82
76
-
The table returned will contain the path to the file, the name of the file, and index of the corresponding HDU, and each FITS header column and value. If two FITS files have distinct columns, they will both appear in the table with `missing` in the appropriate rows.
83
+
The table returned will contain the path to the file, the name of the file, and
84
+
index of the corresponding HDU, and each FITS header column and value. If two
85
+
FITS files have distinct columns, they will both appear in the table with
86
+
`missing` in the appropriate rows.
77
87
78
88
!!! note "Duplicate Keys"
79
-
In certain cases, there are multiple FITS headers with the same key, e.g., `COMMENT`. In these cases, only the first instance of the key-value pair will be stored.
80
-
81
-
If `abspath` is true, the path in the table will be absolute. If `keepext` is true, the name in the table will include the file extension, given by `ext`. `ext` will be used with `endswith` to filter for fits files compatible with `FITSIO.FITS`. `exclude` is a pattern that can be used with `occursin` to exclude certain filenames. For example, to exclude any files containing "sky",
89
+
In certain cases, there are multiple FITS headers with the same key, e.g.,
90
+
`COMMENT`. In these cases, only the first instance of the key-value pair
91
+
will be stored.
92
+
93
+
If `abspath` is true, the path in the table will be absolute. If `keepext` is
94
+
true, the name in the table will include the file extension, given by `ext`.
95
+
`ext` will be used with `endswith` to filter for fits files compatible with
96
+
`FITSIO.FITS`. `exclude` is a pattern that can be used with `occursin` to
97
+
exclude certain filenames. For example, to exclude any files containing "sky",
82
98
```julia
83
99
fitscollection(...; exclude="sky")
84
100
```
85
-
to exclude exact filenames, [regex strings](https://docs.julialang.org/en/v1/manual/strings/#Regular-Expressions-1) will prove powerful
finally, using external tools like [Glob.jl](https://github.com/vtjnash/Glob.jl) allows further customization
107
+
finally, using external tools like [Glob.jl](https://github.com/vtjnash/Glob.jl)
108
+
allows further customization
90
109
```julia
91
110
using Glob
92
111
fitscollection(...; exclude=fn"tek001*.fits") # same as regex match above
93
112
```
94
-
Similarly, `exclude_dir` allows excluding entire folders using pattern matching (e.g. skipping a backup folder `exclude_dir="backup"`).
95
-
`exclude_key` allows excluding certain entries in the header unit of `ImageHDU` in FITS files (e.g. skipping `"HISTORY"` and `""` `exclude_key = ("HISTORY", "")`).
113
+
Similarly, `exclude_dir` allows excluding entire folders using pattern matching
114
+
(e.g. skipping a backup folder `exclude_dir="backup"`).
115
+
`exclude_key` allows excluding certain entries in the header unit of `ImageHDU`
116
+
in FITS files (e.g. skipping `"HISTORY"` and `""` `exclude_key = ("HISTORY", "")`).
96
117
97
-
For more information about the file matching and path deconstruction, see the extended help (`??fitscollection`)
118
+
For more information about the file matching and path deconstruction, see the
119
+
extended help (`??fitscollection`)
98
120
# Extended Help
99
121
100
122
## Parts of a path
101
123
102
-
Let's look at some file paths starting from `"/data"`. Here are examples of how they would be parsed
124
+
Let's look at some file paths starting from `"/data"`. Here are examples of how
125
+
they would be parsed
103
126
104
-
```
127
+
```plain
105
128
root dir base ext
106
129
[----][---][------][---]
107
130
/data/test/tek0001.fits
@@ -111,7 +134,12 @@ Let's look at some file paths starting from `"/data"`. Here are examples of how
111
134
/data/test/sci/tek0001.fits
112
135
```
113
136
114
-
If `keepext` is `true`, `name=base * ext`, otherwise it is just `base`. If `abspath` is `true`, the path will be `root * dir * base * ext`, otherwise it will be `dir * base * ext`. These options allow flexility in creating a table that can be easily saved and loaded to avoid having to manually filter files. Especially consider how `abspath` can allow keeping tables that will transfer easily between computers or between data sources with common structures.
137
+
If `keepext` is `true`, `name=base * ext`, otherwise it is just `base`. If
138
+
`abspath` is `true`, the path will be `root * dir * base * ext`, otherwise it
139
+
will be `dir * base * ext`. These options allow flexility in creating a table
140
+
that can be easily saved and loaded to avoid having to manually filter files.
141
+
Especially consider how `abspath` can allow keeping tables that will transfer
142
+
easily between computers or between data sources with common structures.
115
143
"""
116
144
functionfitscollection(basedir::String;
117
145
recursive =true,
@@ -176,7 +204,8 @@ Iterates over `collection` using each `path` and `hdu` to load data into an `Arr
176
204
collection = fitscollection("~/data/tekdata")
177
205
data = arrays(collection) |> collect
178
206
```
179
-
This returns all image arrays present in `collection`. This can also be used via a for-loop
207
+
This returns all image arrays present in `collection`.
208
+
This can also be used via a for-loop
180
209
```julia
181
210
collection = fitscollection("~/data/tekdata")
182
211
for arr in arrays(collection)
@@ -238,7 +267,8 @@ end
238
267
239
268
Generator for `CCDData`s of entries in data frame.
240
269
241
-
Iterates over `collection` using each `path` and `hdu` to load data into a [`CCDData`](@ref).
270
+
Iterates over `collection` using each `path` and `hdu` to load data into a
271
+
[`CCDData`](@ref).
242
272
243
273
# Examples
244
274
```julia
@@ -271,7 +301,14 @@ end
271
301
272
302
Iterates over the `CCDData`s of the collection applying function `f` at each step.
273
303
274
-
The output from `f` can be saved using the appropriate keyword arguments. The `save_prefix` argument will add a prefix to each filename delimited by `save_delim`. `save_suffix` will add a suffix prior to the extension, which can be manually provided via `ext`, similar to [`fitscollection`](@ref). Files will be saved in the directory they are stored unless `path` is given. Finally, `save` will default to `true` if any of the previous arguments are set, but can be manually overridden (useful for testing). Files will be saved using [`CCDReduction.writefits`](@ref).
304
+
The output from `f` can be saved using the appropriate keyword arguments. The
305
+
`save_prefix` argument will add a prefix to each filename delimited by
306
+
`save_delim`. `save_suffix` will add a suffix prior to the extension, which can
307
+
be manually provided via `ext`, similar to [`fitscollection`](@ref). Files will
308
+
be saved in the directory they are stored unless `path` is given. Finally,
309
+
`save` will default to `true` if any of the previous arguments are set, but can
310
+
be manually overridden (useful for testing). Files will be saved using
311
+
[`CCDReduction.writefits`](@ref).
275
312
276
313
# Example
277
314
```julia
@@ -280,15 +317,17 @@ processed_images = map(ccds(collection)) do img
280
317
trim(img, (:, 1040:1059))
281
318
end
282
319
```
283
-
The above generates `processed_images` which consists of trimmed versions of images present in `collection`.
320
+
The above generates `processed_images` which consists of trimmed versions of
321
+
images present in `collection`.
284
322
285
323
For saving the `processed_images` simultaneously with the operations performed
The trimmed images are saved as `trimmed_(original_name)` (FITS files) at `path = "~/data/tekdata"` as specified by the user.
329
+
The trimmed images are saved as `trimmed_(original_name)` (FITS files) at
330
+
`path = "~/data/tekdata"` as specified by the user.
292
331
"""
293
332
functionccds(f,
294
333
collection;
@@ -332,7 +371,14 @@ end
332
371
333
372
Iterates over the file paths of the collection applying function `f` at each step.
334
373
335
-
The output from `f` can be saved using the appropriate keyword arguments. The `save_prefix` argument will add a prefix to each filename delimited by `save_delim`. `save_suffix` will add a suffix prior to the extension, which can be manually provided via `ext`, similar to [`fitscollection`](@ref). Files will be saved in the directory they are stored unless `path` is given. Finally, `save` will default to `true` if any of the previous arguments are set, but can be manually overridden (useful for testing). Files will be saved using [`CCDReduction.writefits`](@ref).
374
+
The output from `f` can be saved using the appropriate keyword arguments. The
375
+
`save_prefix` argument will add a prefix to each filename delimited by
376
+
`save_delim`. `save_suffix` will add a suffix prior to the extension, which can
377
+
be manually provided via `ext`, similar to [`fitscollection`](@ref). Files will
378
+
be saved in the directory they are stored unless `path` is given. Finally,
379
+
`save` will default to `true` if any of the previous arguments are set, but can
380
+
be manually overridden (useful for testing). Files will be saved using
381
+
[`CCDReduction.writefits`](@ref).
336
382
337
383
# Examples
338
384
```julia
@@ -344,7 +390,8 @@ data = map(filenames(collection)) do path
344
390
data
345
391
end
346
392
```
347
-
The above generates `data` which consists of image arrays corresponding to 1st hdu of FITS file paths present in `collection`.
393
+
The above generates `data` which consists of image arrays corresponding to 1st
394
+
hdu of FITS file paths present in `collection`.
348
395
For saving the `data` simultaneously with the operations performed
349
396
```julia
350
397
data = map(filenames(collection; path = "~/data/tekdata", save_prefix = "retrieved_from_filename")) do img
The retrieved data is saved as `retrieved_from_filename_(original_name)` (FITS files) at `path = "~/data/tekdata"` as specified by the user.
404
+
The retrieved data is saved as `retrieved_from_filename_(original_name)`
405
+
(FITS files) at `path = "~/data/tekdata"` as specified by the user.
358
406
"""
359
407
functionfilenames(f,
360
408
collection;
@@ -398,7 +446,14 @@ end
398
446
399
447
Iterates over the image arrays of the collection applying function `f` at each step.
400
448
401
-
The output from `f` can be saved using the appropriate keyword arguments. The `save_prefix` argument will add a prefix to each filename delimited by `save_delim`. `save_suffix` will add a suffix prior to the extension, which can be manually provided via `ext`, similar to [`fitscollection`](@ref). Files will be saved in the directory they are stored unless `path` is given. Finally, `save` will default to `true` if any of the previous arguments are set, but can be manually overridden (useful for testing). Files will be saved using [`CCDReduction.writefits`](@ref).
449
+
The output from `f` can be saved using the appropriate keyword arguments.
450
+
The `save_prefix` argument will add a prefix to each filename delimited by
451
+
`save_delim`. `save_suffix` will add a suffix prior to the extension, which can
452
+
be manually provided via `ext`, similar to [`fitscollection`](@ref). Files will
453
+
be saved in the directory they are stored unless `path` is given. Finally,
454
+
`save` will default to `true` if any of the previous arguments are set, but can
455
+
be manually overridden (useful for testing). Files will be saved using
456
+
[`CCDReduction.writefits`](@ref).
402
457
403
458
# Examples
404
459
```julia
@@ -407,14 +462,16 @@ processed_images = map(arrays(collection)) do arr
407
462
trim(arr, (:, 1040:1059))
408
463
end
409
464
```
410
-
The above generates `processed_images` which consists of trimmed versions of image arrays present in `collection`.
465
+
The above generates `processed_images` which consists of trimmed versions of
466
+
image arrays present in `collection`.
411
467
For saving the `processed_images` simultaneously with the operations performed
Copy file name to clipboardExpand all lines: src/fits.jl
+6-2Lines changed: 6 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -5,8 +5,12 @@
5
5
6
6
Loads the given HDU as an `Array`, permuting the dimensions appropriately.
7
7
8
-
FITSIO.jl takes over memory read in by cfitsio, which reads in row-major form, whereas when Julia takes that memory, it is assumed as column major.
9
-
Therefore all data read by [`FITSIO.read`](http://juliaastro.github.io/FITSIO.jl/latest/api.html#Base.read-Tuple{ImageHDU}) is transposed. This function allows the user to read data in a consistent way to `Array` by transposing after reading.
8
+
FITSIO.jl takes over memory read in by cfitsio, which reads in row-major form,
9
+
whereas when Julia takes that memory, it is assumed as column major.
0 commit comments