Argus file standard

[AlvaroParker]

Hi! Looking on the repo and on the website I couldn't find a standard on the file format produced by the argus binary executable. Is there a place to get it if it's exists? If not, where can i learn what's the expected argus file format that tools like argus-ra expect.

I notice that when sending the argus output to a udp socket, the first time argus is started, it always sends the same 128 binary data, so I would expect that to be the header, however I couldn't find docs about it.

[openargus]

Hey Alvaro, The format is described in the ./include/argus_out.h and ./include/argus_defs.h files using the IETF style of data record specification … These may not be complete, but they cover 90% of the data structure definitions that haven't changed in a while … The way it works … Argus output is a stream of TLV based records (type, length value) … All have a 4 byte header that identifiers the type, subtype, options and length. There are 2 types of record header formats, identified by the subtype, which indicate whether there is an 8-bit or 16-bit length. There are 4 types of Argus records Management (MAR) Flow (FAR) Event (EAR) Supplement (SAR) All argus data records start with a 128 byte Argus Management Record (MAR). This has a unique pattern that /etc/magic can use to identify argus files which is included in the clients distribution. All streams start with a 128 byte MAR START record, end with a 128 bytes MAR END record and there are periodic 128 byte MAR STATUS records, which have internal stats in them. These act as stream identifiers, keep alive records and stream recovery tokens. There are many, many flow records (FAR), each are documented in the ./include/argus_defs.h and ./include/argus_out.h files in either argus or clients repo. The Argus Event (EAR) record currently is an XML structured metadata record that can contain anything you want. The format is documented in the ./include/argus_defs.h file and has the 4-byte header, and some structs for data. The Supplement record is not being used today. Hopefully this will help you. I highly recommend that you use the clients library to read and write argus data streams, as it has compress and encryption support that will make life difficult if you try to implement without its help. The ratemplate.c example in the client distro is there to help you build your own argus processing engines, and all of the client programs are there as examples. Carter

…

On Nov 6, 2024, at 12:05 PM, Alvaro Parker ***@***.***> wrote: Hi! Looking on the repo and on the website I couldn't find a standard on the file format produced by the argus binary executable. Is there a place to get it if it's exists? If not, where can i learn what's the expected argus file format that tools like argus-ra expect. I notice that when sending the argus output to a udp socket, the first time argus is started, it always sends the same 128 binary data, so I would expect that to be the header, however I couldn't find docs about it. — Reply to this email directly, view it on GitHub <#15>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3MTCC23VI7NEKFMWNJRHDZ7JD4DAVCNFSM6AAAAABRJJWVBGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYZTQNZQG4ZDQNQ>. You are receiving this because you are subscribed to this thread.

[AlvaroParker]

Hey @openargus

Thanks for that detailed response. The reason I was looking for the format is because I'm reading the data generated by argus over a UDP socket server, however because of limitations of my application and other reasons, I have to read the data for a specific amount of time, then use argus-ra to convert it to csv and then keep reading from the UDP socket from a cleared buffer, meaning I lost the header which was on the first chunk, which causes argus-ra to interpret the next chunks as not a valid argus data, which in turns make the ra command to fail.

Seems like the solution would be to save the header, but I wasn't sure of the length of it. I was able to see that the first packet was always 128 bits long but I wasn't sure if that was the header. My UDP server is written in C#, are there any alternative besides manually invoking the ra command for parsing the output generated by argus?

[openargus]

We have tools that do this … at least from what I gather from your description. Checkout rabins.1 Argus can write to UDP sockets, and all the clients can read argus records from UDP sockets … So you should be able to do what you’ve described without any coding … Reading from a UDP socket … flushing output as csv periodically … lets say every 60 seconds. Use the -B option to buffer output so that you wait long enough to get all the records for the time bin. % rabins -S argus-udp://1.2.3.4:5678 -B 5s -M time 60s -c, rabins.1 is an argus record aggregator, so you can do a lot of processing on the records before you write them out. And rabins.1 has a lot of options that are kind of esoteric, until you get into the issues with this type of processing … What do you think ?? Carter

…

On Nov 6, 2024, at 3:26 PM, Alvaro Parker ***@***.***> wrote: Hey @openargus <https://github.com/openargus> Thanks for that detailed response. The reason I was looking for the format is because I'm reading the data generated by argus over a UDP socket server, however because of limitations of my application and other reasons, I have to read the data for a specific amount of time, then use argus-ra to convert it to csv and then keep reading from the UDP socket from a cleared buffer, meaning I lost the header which was on the first chunk, which causes argus-ra to interpret the next chunks as not a valid argus data, which in turns make the ra command to fail. Seems like the solution would be to save the header, but I wasn't sure of the length of it. I was able to see that the first packet was always 128 bits long but I wasn't sure if that was the header. My UDP server is written in C#, are there any alternative besides manually invoking the ra command for parsing the output generated by argus? — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3MTCFTZUOULLWZW25S6A3Z7J3PLAVCNFSM6AAAAABRJJWVBGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRQG4YTANZZGM>. You are receiving this because you were mentioned.