v2.0.5

docs/ObservationProperties.md

@@ -46,7 +46,7 @@
   Cal Bench filter 1 position. Throughput = 10^-OD. Values: OD 0.1, OD 1.0, OD 1.3, OD 2.0, OD 3.0, OD 4.0
 
 **CalND2**: `str`
-  Cal Bench filter 2 position. Throughput = 10^-OD. Values: OD 0.1, OD 0.3, OD 0.5, OD 0.8, OD 1.0, OD 4.0
+  Cal Bench filter 2 position. Throughput = 10^-OD. Values: OD 0.1, OD 0.3, OD 0.5, OD 1.0, OD 1.3, OD 2.0
 
 **NodN**: `float`
   [arcseconds] Distance to nod the telescope North before starting exposure.

docs/announcements/2026-02-01_26A-status.md

@@ -0,0 +1,23 @@
+# Semester 26A Stability Announcement
+
+Precision radial velocity measurements which are comparable across long time scales (months to years) require a stable, well calibrated instrument. KPF has had several problems in the past which have impacted stability and calibrations and these will likely impact data taken in the 26A semester.
+
+We discuss a few details below, but the high level summary is that we currently do not have confidence in the instrument sub-systems to maintain temperature stability and provide reliable calibrations.  As a result, users of KPF who have science goals which require long term stability of the instrument should take this into account when choosing which science programs to execute with their time in 26A.
+
+In addition, the KPF DRP is not performing well on faint object extractions.  Somewhere around Gmag of 15-16 is where the performance degrades significantly, so we advise users to avoid faint stars if they need immediate results.  We do expect the DRP to handle this case properly in the long term, so data taken now will likely be useful in the future, but the results coming out at the moment will significantly underperform the ETC.
+
+## Temperature Stability
+
+The most impactful stability factor for KPF has been the detector temperatures. In order for RV measurements at different times to be compared, the system must not change temperature significantly during that time span as the hysteresis induced by a change and return to temperature results in a zero point change for the RV measurements.  The goal is for the system to be stable at the ~1 mK level, while changes of ~1 K are almost certain to break the RV zero point.
+
+The Green and Red CCDs in the main spectrometer are likely to experience thermal events of a few degrees C (or more) above the -100 C detector setpoints lasting a few hours. While the number and timing of these are difficult to predict, it is likely that KPF will not maintain long-term RV stability over 2026A. Thus, time series RV measurements over long periods (weeks or months) are likely to be compromised by changing RV zero points at the ~10 m/s level. Experience has shown that these zero-point changes are difficult/impossible to calibrate out and are different for each star observed. Science projects that rely on a time series of KPF RVs or spectra over a short timescale (a night to a few nights) are unlikely to be affected by the warmups (which would have to occur during the observing sequence to be a problem). Users who have science goals which require high precision (a few m/s or better) on long time scales should take this into account when choosing which science programs to execute with their time in 26A.
+
+The WMKO and KPF Build Teams are currently planning for an additional servicing mission to address the cooling problems, however that will take time to plan and implement. 
+
+## Calibration Reliability
+
+In addition to stability, the instrument also requires frequent, high precision calibrations.  The most fundamental calibrator for KPF is the Laser Frequency Comb (LFC).  The LFC has had periods of significant unreliability and the wavelength coverage for the LFC has been inconsistent. This has resulted in long periods of poor calibrations which requires bootstrapping the wavelength solution from other calibrators or over longer time periods which impacts RV measurement precision.
+
+## Detector Noise
+
+In addition to the stability concerns, KPF has also faced challenges with detector noise.  The elevated read noise present since early 2024 has been partially mitigated.  Currently, the Red detector is performing at a nominal level (4.3 electrons read noise) and although the Green detector noise has been significantly reduced from the 25-35 electrons seen in the past, it remains elevated at about 10 electrons. An intervention to address this is being planned.

docs/buildingOBs.md

@@ -10,18 +10,18 @@ Observers can create OBs in 3 ways:
 
 The data in an OB can be divided in to three categories:
 
-**Target**: The OB will contain information about the target beyond what is in a typical Keck Star List entry in order to flow that information to the FITS header and the data reduction pipeline (DRP).  The target section is only needed if the OB has observations (i.e. it is not purely a calibration OB). Here is a description of all [Target Properties](../TargetProperties).
+**Target**: The OB will contain information about the target beyond what is in a typical Keck Star List entry in order to flow that information to the FITS header and the data reduction pipeline (DRP).  The target section is only needed if the OB has observations (i.e. it is not purely a calibration OB). Here is a description of all [Target Properties](TargetProperties.md).
 
-**Calibrations**: An OB can contain calibrations, these are not typically used by the observer (slewcals are handled separately). Here is a description of all [Calibration Properties](../CalibrationProperties). The Calibrations section of an ON is a list of Calibration entries.
+**Calibrations**: An OB can contain calibrations, these are not typically used by the observer (slewcals are handled separately). Here is a description of all [Calibration Properties](CalibrationProperties.md). The Calibrations section of an ON is a list of Calibration entries.
 
-**Observations**: Finally, the OB will contain a list of observations to be made of the target. For typical KPF observers, this will only have one entry, but multiple entries are supported. Each entry describes a set of exposures on the target and contains the information on how those exposures should be executed. Here is a description of all [Observation Properties](../ObservationProperties). The Observations section of an ON is a list of Observation entries.
+**Observations**: Finally, the OB will contain a list of observations to be made of the target. For typical KPF observers, this will only have one entry, but multiple entries are supported. Each entry describes a set of exposures on the target and contains the information on how those exposures should be executed. Here is a description of all [Observation Properties](ObservationProperties.md). The Observations section of an ON is a list of Observation entries.
 
 Note that not all properties are needed in every case. For example, an observation with `ExpMeterMode: 'monitor'` will not need values for `ExpMeterBin` and `ExpMeterThreshold`.
 
 
 ## Example On Sky Science OB
 
-This is an example of what the text file form of an OB might look like. The file is a  `yaml` format which resolves in to a python dict with keys for "Target", "Calibrations" and "Observations" (not all are required).  The "Target" entry is a dict with the various [Target Properties](../TargetProperties).  The "Calibrations" entry (if present) is a **list** of dictionaries, each with the various [Calibration Properties](../CalibrationProperties).  Similarly, the "Observations" entry is a **list** of dictionaries, each with the various [Observation Properties](../ObservationProperties).
+This is an example of what the text file form of an OB might look like. The file is a  `yaml` format which resolves in to a python dict with keys for "Target", "Calibrations" and "Observations" (not all are required).  The "Target" entry is a dict with the various [Target Properties](TargetProperties.md).  The "Calibrations" entry (if present) is a **list** of dictionaries, each with the various [Calibration Properties](CalibrationProperties.md).  Similarly, the "Observations" entry is a **list** of dictionaries, each with the various [Observation Properties](ObservationProperties.md).
 
 The example below has a Target, no Calibrations, and a single Observaton:
 

docs/status.md

@@ -1,25 +1,48 @@
 # Instrument Status
 
-### Current and Past Announcements
+### Current Announcements
 
-* 2025 August: [26A Stability Announcement](KPF Stability Statement - August 15 2025.pdf)
-* 2023 September: [Keck Science Meeting presentation](Keck Science Meeting 2023 Breakout Session.pdf)
+* 2026 February: [26A Status Announcement](announcements/2026-02-01_26A-status.md)
 
 ### Status Summary by Subsystem
 
 This is an attempt to summarize the status of various sub-systems of the instrument.  Each sub-system name is color coded to indicate the status at a glance: <font color="green">green</font> means functioning normally, <font color="orange">orange</font> means mostly normal, but with some caveats or minor issues, and <font color="red">red</font> means the sub-system is compromised in some way.
 
-Last Status Update: 2025-10-22
-
-- **<font color="orange">Detector Noise</font>**: Starting in November of 2024, additional pattern noise has been present on the detectors.  We have been working on eliminating the spurious nose, but we have been unable to completely remove it.  As of late-October 2025 the read noise on the Green side remains elevated (~10 electrons), while the red side is near our nominal target (~4.3 electrons).
-- **<font color="orange">LFC</font>**: We are evaluating the reliability of the LFC after recent service. Initial indications look prominsing on reliability, though the bluest flux (below ~490 nm) is not consistent.
+- **<font color="orange">Detector Noise</font>**: Starting in November of 2024, additional non-gaussian noise has been present on the detectors. As of late-October 2025 the read noise on the Green side remains elevated (~10 electrons), while the red side is at our target level (~4.3 electrons).
+- **<font color="orange">LFC</font>**: Initial evaluations of the reliability of the LFC after the recent service look promising, though the bluest flux (below ~490 nm) is not consistent.
+- **<font color="red">Detector Cooling Systems</font>**: The green side CCR has  very little overhead on maintaining temperature and has quasi-periodic deviations which affect the detector. Red side is performing well, but has shown evidence of a slow degradation of performance.
 - **<font color="green">Etalon</font>**: Operational.
-- **<font color="red">Detector Cooling Systems</font>**: Both detectors are now cooled with closed cycle refrigerators (CCRs). The green side CCR has  problems and has very little overhead on maintaining temperature and has quasi-periodic deviations which affect the detector.  Red side is performing well.
-- **<font color="green">Detector Errors</font>**: The red and green detectors suffer from occasional “start state errors” in which the affected detector remains in the start phase and does not produce a useful exposure. The observing scripts detect this, abort the current exposure (with read out) and start a fresh exposure on both cameras. **No action is necessary on the part of the observer.**  The occurrence rate is such that around one in every 180 exposures is affected by one of the two detectors experiencing this error.
+- **<font color="green">Detector Errors</font>**: The red and green detectors suffer from occasional “start state errors” in which the affected detector does not produce a useful exposure. The observing scripts detect this, abort the exposure (with read out) and start a fresh exposure on both cameras. **No action is necessary on the part of the observer.**  The occurrence rate is such that around one in every 180 exposures is affected by one of the two detectors experiencing this error.
 - **<font color="green">Ca H&K Detector</font>**: The CA H&K detector is operational.
 - **<font color="green">Exposure Meter Terminated Exposures</font>**: Operational.
 - **<font color="green">Tip Tilt Corrections</font>**: The tip tilt axis are currently correcting as expected.
 - **<font color="green">Double Star Observations</font>**: Operational.
 - **<font color="green">Simultaneous Calibration (SimulCal)</font>**: Simultaneous calibrations are supported.
-- **<font color="orange">Nod to Sky Observations</font>**: For observations which need a sky measurement other than the built in sky fibers, nodding away to a sky position can be accomplished manually by running separate OBs for the target and sky and asking the OA to offset the telescope as appropriate.  We plan to build a separate Nod To Sky observing mode which will accomplish this within a single OB, but that is not yet available.
 - **<font color="red">Off Target Guiding</font>**: Not yet commissioned.  Currently, the tip tilt system must be able to detect the science target in order to position it on the fiber.
+
+Last Updated: 2026-02-01
+
+### KPF Era 4.0 Temperature Stability Summary
+
+KPF Era 4.0 began in late-October 2025 after Servicing Mission 4. We continue to have issues with the cooling systems for the detectors, especially the Green side.  We list below all temperature excursions in which one of the detectors deviated temperature by more than 5 mK from the set point.  Excursions of order 1 K (1000 mK) or more are expected to induce a radial velocity offset which is not calibratable.  We are providing this data as a guide, but users should not assume that past performance is a good indicator of future performance -- we have seen indications that the cooling systems are slowly degrading, so an increasing rate of these temperature excursions is a distinct possibility.
+
+| Side  | Duration     | Delta T   | Start (HST)         | End (HST)           |
+| ----- | ------------ | --------- | ------------------- | ------------------- |
+| Green |   0.05 hours |      7 mK | 2025-11-09 21:22:14 | 2025-11-09 21:25:06 |
+| Green |   0.36 hours |    109 mK | 2025-11-10 05:37:12 | 2025-11-10 05:58:50 |
+| Green |   2.78 hours |  2,825 mK | 2025-11-10 11:09:06 | 2025-11-10 13:55:54 |
+| Green |   0.63 hours |    250 mK | 2025-11-15 11:28:36 | 2025-11-15 12:06:32 |
+| Green |   2.54 hours |  5,779 mK | 2025-11-16 14:27:20 | 2025-11-16 16:59:54 |
+| Green |   4.04 hours |  6,147 mK | 2025-11-17 07:52:06 | 2025-11-17 11:54:40 |
+| Green |   2.31 hours |  4,200 mK | 2025-11-18 03:40:28 | 2025-11-18 05:59:16 |
+| Green |   3.14 hours |  5,862 mK | 2025-11-18 23:08:20 | 2025-11-19 02:16:52 |
+| Green |   0.56 hours |     95 mK | 2026-01-23 21:12:46 | 2026-01-23 21:46:16 |
+| Green |   1.20 hours |    712 mK | 2026-01-25 15:25:34 | 2026-01-25 16:37:48 |
+| Green |   0.46 hours |    129 mK | 2026-01-30 19:36:24 | 2026-01-30 20:03:54 |
+| Green |  13.82 hours | 53,527 mK | 2026-02-17 21:17:12 | 2026-02-18 11:06:40 |
+| Green |   4.48 hours | 11,585 mK | 2026-02-19 02:40:56 | 2026-02-19 07:09:32 |
+
+### Past Announcements
+
+* 2025 August: [26A Stability Announcement](KPF Stability Statement - August 15 2025.pdf)
+* 2023 September: [Keck Science Meeting presentation](Keck Science Meeting 2023 Breakout Session.pdf)

docs/troubleshooting.md

@@ -7,6 +7,8 @@
 * Scripts
     * [Existing Script is Running](#existing-script-is-running)
     * [Agitator Use is Disabled](#agitator-use-is-disabled)
+    * [Start of Night Script Failed](#start-of-night-script-failed)
+    * [End of Night Script Failed](#end-of-night-script-failed)
 * Calibrations
     * [Calibration Source is Not Working](#calibration-source-is-not-working)
     * [SlewCal or Simultaneous Calibration Source is Wrong](#slewcal-or-simultaneous-calibration-source-is-wrong)
@@ -32,6 +34,7 @@
 * SoCal
     * [Enclosure Lid Does not Move 1](#enclosure-lid-does-not-move-1)
     * [Enclosure Lid Does not Move 2](#enclosure-lid-does-not-move-2)
+    * [EKO Sun Tracker is not on Sun](#eko-sun-tracker-is-not-on-sun)
 
 
 # General Principles
@@ -109,6 +112,75 @@ This means that the `kpfconfig.USEAGITATOR` keyword is set to “No”.  This ke
 
 The agitator can be reenabled by simply setting the keyword to “Yes”.  **This should only be done by WMKO staff** and should only be done if the agitator is fully functional.  A broken or misbehaving agitator mechanism presents a significant danger to the science fibers.
 
+## Start of Night Script Failed
+
+<u>Symptom</u>:
+
+When executing the start of night script, the script failed to read and set AO keywords.
+
+<u>Problem</u>:
+
+There is some kind of AO gateway communicaiton problem, and so far we don't know what the root cause is. This happens intermittently. 
+
+<u>Solution</u>:
+
+Modify AO keywords as k1obsao in a k1aoserver-new terminal.
+
+Open AO hatch and check status:
+
+```
+modify -s ao aohatchcmd=open
+show -s ao aohatchsts
+```
+
+Send PCU to KPF:
+
+```
+modify -s ao pcuname=KPF 
+show -s ao pcuname
+```
+
+Send AO rotator to 0 deg:
+
+```
+modify -s ao obrt=0 
+show -s ao obrt
+```
+
+Set rotator to stationary:
+
+```
+modify -s dcs rotmode=stationary
+show -s dcs rotmode
+```
+
+## End of Night Script Failed
+
+<u>Symptom</u>:
+
+When executing the end of night script, the script failed to read and set AO keywords.
+
+<u>Problem</u>:
+
+There is some kind of AO gateway communicaiton problem, and so far we don't know what the root causee is. This happens intermittently. 
+
+<u>Solution</u>:
+
+Modify AO keywords as k1obsao in a k1aoserver-new terminal.
+
+Close AO hatch and check status:
+
+```
+modify -s ao aohatchcmd=close
+show -s ao aohatchsts
+```
+
+Send AO rotator to 45 deg:
+
+```
+modify -s ao obrt=45 
+show -s ao obrt
+```
 
 # Calibrations
 
@@ -557,3 +629,20 @@ The `~/grep_for_dome_error` script will exclude many of the noisy, not useful li
 <u>Solution</u>:
 
 Reboot the controller (raspberry pi): `sudo reboot` and restart the kpfsocal3 dispatcher: `kpf restart kpfsocal3`.  Multiple reboots may be required.
+
+## EKO Sun Tracker is not on Sun
+
+<u>Symptom</u>:
+
+The EKO solar tracker is not pointed at the sun, it may be parked.
+
+<u>Problem</u>:
+
+The tracker is not in the right mode.
+
+<u>Solution</u>:
+
+* Run `modify -s kpfsocal EKOCMD=2` which tells EKO to "guide" on the Sun.
+* Run `modify -s kpfsocal EKOMODE=3` which sets the EKO in to "Sun-sensor with Learning" mode.
+
+If that fails, restart the EKO dispatcher: `kpf restart kpfsocal1` and run the commands again.