ldmsend_nws.c

[dgaer]

Rewrite of ldmsend.c that will allow send retries if an error occurs during the file upload. Add an optional notification to verify that the product was added to the LDM queue on the remote host. Due to some network issues, several of our sites has product upload failures using ldmsend. As some critical products are only sent at specific times we needed a way to resend the product during periods when the network was saturated. Due to our firewall configurations we cannot request from our downstream servers so we could not do a pqinsert at the site and request from the downstream queue. Ldmsend works most of the time but was failing during periods of when the network was saturated. The new version of ldmsend allow us to resend and verify the upload, for example:

su - ldm
/bin/bash
dd if=/dev/urandom of=/tmp/testfile.1 bs=1024 count=8192
cd /tmp
status=$(/usr/local/ldm/util/ldmsend_nws -vxnl- -h 216.38.81.25 -f EXP -o 3600 -r 5 -R 60 -T 25 -p '^testfile.*' testfile.1)
echo $status

NOTE: For testing purposes the new version of ldmsend was compiled as ldmsend_nws and copied to the util directory.

If status variable equals PASS the file made is to the upstream LDM queue. If status equals anything else, the file did not upload and our scripts can be adjusted to send alert messages to the forecasters if any critical upload failed.

[semmerson]

ldmsend_nws(1) calls forn5(). The purpose of ldmsend(1) is to act as an upstream LDM. Why does ldmsend_nws(1) act as a downstream LDM?

[semmerson]

Do your firewalls allow TCP connections to port 80?

[dgaer]

Hi Steve,

With the exception of our public Web server, all access public to port
80 is blocked. Access to our LDM queue monitor on port 80 is limited to
our NWS field offices.

Thanks,
Doug

On 09/12/2016 06:40 PM, Steven Emmerson wrote:

Do your firewalls allow TCP connections to port 80?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#42 (comment), or
mute the thread
https://github.com/notifications/unsubscribe-auth/AOHr03hzjazjntY8-jXaUwonyCxf3m7wks5qpeLygaJpZM4GD_tA.

// ****************************************** //
// Douglas Gaer
// Information Technology Officer
// Linux Engineer/Administrator/Programmer
//
// Email: Douglas.Gaer@noaa.gov
// Phone: 1-682-703-3714
//
// National Weather Service
// SRH/STSD/Technology Infusion Branch
// 819 Taylor Street, Room 10A06
// Fort Worth, TX 76102
//
// http://www.srh.noaa.gov
// ****************************************** //

[dgaer]

Hi Steve,

The reason I added the NOTIFYME call was to verify the product was
inserted into the upstream LDM send. For example, all the NWS coastal
offices upload a forecaster generated wind grid that initiates an on
demand wave model run. Due to various network issues and other system
issues there are times when the LDM send does not successfully deliver
the wind grid and we have no indication the wind grid was not delivered
to the upstream queue. Due to various network and systems setting we
cannot request the product from the downstream queue, leaving ldmsend as
our only option. In ldmsend_nws, a NOTIFYME runs to verify the product
made it into the upstream queue. If the product upload fails, the upload
script will detect the upload failure and send an AWIPS WRK product to
alert the forecasters their wind grid was not uploaded. At this point
the forecasters can initiate another model run. The majority of our LDM
send uploads are observation data. Given the frequency of the
observation data, if one or two product uploads fail due to a network
error, we may never notice. But there are a few programs where the every
data upload has to work or signal a failure event.

Thanks,
Doug

On 09/12/2016 06:36 PM, Steven Emmerson wrote:

|ldmsend_nws(1)| calls |forn5()|. The purpose of |ldmsend(1)| is to
act as an upstream LDM. Why does |ldmsend_nws(1)| act as a downstream LDM?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#42 (comment), or
mute the thread
https://github.com/notifications/unsubscribe-auth/AOHr04wT9AIMWWNYDR2N7SNsYni2y1rIks5qpeILgaJpZM4GD_tA.

// ****************************************** //
// Douglas Gaer
// Information Technology Officer
// Linux Engineer/Administrator/Programmer
//
// Email: Douglas.Gaer@noaa.gov
// Phone: 1-682-703-3714
//
// National Weather Service
// SRH/STSD/Technology Infusion Branch
// 819 Taylor Street, Room 10A06
// Fort Worth, TX 76102
//
// http://www.srh.noaa.gov
// ****************************************** //

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}