(g)ULP!
Loading...
Searching...
No Matches
GUiLP

Brief usage / porting guide

Login and operation selection

User is presented with a prompt asking the address of the gulp instance and credentials.

1.png

The actual stand-alone client also allows choosing a local session, i.e. without connecting to the backend (when it started as an internal tool it had basic processing features that have now been moved on server side). /login

After logging in, an “Operation selection” window allows the user to choose the operation to work on (operation = investigation / incident). /operation_list If we will eventually decide to support different Elastic search indexes, a combined index/operation selection window should allow to choose index first, then list the operations for the selected index.

2.png

Once the operation has been selected, the client will list the saved sessions if any. In case there is no saved working session to restore, or the user wants to start a new one, the client will first interrogate the server in order to know the first and last event timestamp for the entire operation, and help the user decide an appropriate time window to work on (especially when the data on the server spans on multiple months or even years – the actual check is). This window is shown if the time span between the oldest and newest event is more 30 days.

/query_max_min

The range selection allows to choose between some presets (last day / week / month / full range) or manually specify the time range.

3.png

The next step is to choose which contexts/sources to use in a session, further in skimming the amount of data and thus narrow the focus.

4.png

In case the user has chosen to load a saved session, the saved time range and list of source will be just loaded from the saved session data.

Main window

[IMAGE GOES HERE]

The main window is divided into 3 main areas: Context/source list, Events, Event details.

Some clarification about the lexicon: we use the word “source” to refer to a log source (e.g. a single log file on a system) and the word “context” to describe the system to which the source belongs (i.e. the system where the log originated)

Context/source list

A single source shows information such as the name of the log, the context it belongs to, the total number of events. It actually looks like this:

5.png

There’s plenty of space to add support to other additional information like custom tags and so on. The 6 menu items on the right allow to manually arrange the source, pin, hide/show it, set a custom filter or access an additional menu with further items such as undo last operation (last query), change rendering type, cancel any ongoing operation (i.e. query) for the currently selected source. The additional menu looks like this:

6.png

‍[!NOTE] We’re not yet sure if we want to keep things as they are, or arrange things in a sort of “tree” allowing to group sources belonging to the same context… suggestions or proposals are welcome.

The source list is populated by calling /query_operations with token and index in the querystring, and then iterating the results, taking into account only those with the right operation_id. (@Valerio a more appropriate api for querying a single operation would be better).

Data in the response looks like this:

{
"status": "success",
"timestamp_msec": 1720121934425,
"req_id": "faa1947f-78c7-4d5c-b016-e69155c6cba3",
"data": [
{
"name": "Operation-name",
"id": 1,
"contexts": [
{
"name": "Context 1",
"doc_count": 457483,
"plugins": [
{
"name": "win_evtx",
"src_file": [
{
"name": "Security.evtx",
"doc_count": 202947,
"max_event.code": 5061,
"min_event.code": 4616,
"min_@timestamp": 1717147858352,
"max_@timestamp": 1717488873202
},
{
"name": "DNS Server.evtx",
"doc_count": 92718,
"max_event.code": 7693,
"min_event.code": 2,
"min_@timestamp": 1634218860636,
"max_@timestamp": 1717487957421
}
]
}
]
},
{
"name": "Context 2",
"doc_count": 457483,
"plugins": [
{
"name": "apache_clf",
"src_file": [
{
"name": "webserver.log",
"doc_count": 23351,
"max_event.code": 499,
"min_event.code": 200,
"min_@timestamp": 1693365081000,
"max_@timestamp": 1693386812000
}
]
}
]
}
]
}
]
}

Every context in the operation has a list of plugins (representing the ingestion plugin, and thus the type of events to be represented). For each plugin a list of source files is given, with the “local” max/min timestamps and max/min event codes (more info on this are found later in the document).

Main event window

The main part shows the events in each source. On the upper part a ruler shows the nearest time chunks, with variable tick values (depending on the zoom level) such as seconds, minutes, hours, days, months etc. Actual resolution gets as high as millisecond. Further evolution should take into account supporting nanosecond resolution.

7.png

The rest of the space is occupied by the events.

8.png

General Metadata

The violet full-height line on top of everything shows the timestamp corresponding to the mouse position in the window.

9.png

The yellow and orange vertical stripes (on the left and right bounds) mark the time frame originally selected by the user.

10.png

Row Metadata

The double green vertical lines in rows show the bounds for the single source, i.e. the timestamp corresponding to the first and last event present in the dataset on the backend. This is intended to be a metadata, shown also if the events are filtered out or are not in the selected timeframe. The height is a bit more than the row itself, so not to confuse this indicator with actual events. Next to this indicator, the total number of events (available) is shown in gray, while the amount of events actually on screen is shown in blue. This comes handy to see whether data is being filtered, and how many events are shown by the selected filter.

11.png

Another indicator is a dashed line at the center of the row, when such a line is shown it means that the row is still receiving data from the back-end.

12.png

Row data

Each row shows the events and notes for the corresponding source. On a row both events and notes can be displayed: events are drawn in full height, while notes are actually drawn only in the upper half.

13.png

Events

Events for each row are retrieved using /query_gulp with a GulpQueryFilter to match the right src/context and a GulpQueryOptions object to ask for results in descending order (we are usually interested in newer events first):

  • GulpQueryFilter → start = session_mintime, end = session_maxtime, operationId = operationi,; context = current_context, srcfile = current_sourcefile.
  • GulpQueryOptions → sort = "@timestamp", "desc"

The response has a “CollabStats” object (whose name is misleading and should be changed soon) with a list of events with basic properties in the “query_results” array:

{
"status": "success",
"data": {
"id": null,
"type": 6,
"req_id": "639432bd-db7f-4a7f-a226-400a77af4c92",
"operation_id": null,
"client_id": null,
"context": null,
"status": 1,
"time_created": 1720124695866,
"time_expire": 0,
"time_update": 0,
"time_end": 1720124696005,
"ev_failed": 0,
"ev_skipped": 0,
"ev_processed": 0,
"files_processed": 0,
"files_total": 0,
"queries_total": 1,
"sigma_group_results": null,
"query_results": [
{
"events": [
{
"operation_id": 6,
"@timestamp": 1717147409839,
"source.domain": "test_uploader_context",
"log.file.path": "Security.evtx",
"event.code": "4797",
"event.duration": 0,
"_id": "8c15b46dc6f5259a92d4dad911b737ed529f7f4fe5330b79255863e048b0fc6e46b603694dee987aa64193a774f8dec33f36e72dcb02a667a2a7cf3b29b0a7b9"
},
{
"operation_id": 6,
"@timestamp": 1717147409834,
"source.domain": "test_uploader_context",
"log.file.path": "Security.evtx",
"event.code": "4797",
"event.duration": 0,
"_id": "1193047aacd499d1cece1fc2122c7e2ff2a39916006e694abe349e04e5b9a4ad9ff6bc7a964f4cfe74e1c00149f43aebe9dc00c67280e6db11cd0159e7cdb08e"
},
{
"operation_id": 6,
"@timestamp": 1717147409820,
"source.domain": "test_uploader_context",
"log.file.path": "Security.evtx",
"event.code": "4797",
"event.duration": 0,
"_id": "82322ecd3f4a25f9a40b2d020178530de7900735b03294e8b30ca196a5909fe18994c874e44fbd9a80a7547172b1f4f35d840509997c3bef2dcf66a57327c095"
}
],
"aggregations": null,
"total_hits": 9037,
"search_after": [
1717419026242, "d0e7a4fac93fcbf728c2e352248a2bf76821c3449e8ab93b0e64f7146e7a288947635bb3b8ff814e6e139a4ee5a23b53efb8688566252adf5a90dd6be47f70fe"
],
"query_glyph_id": null,
"error": null,
"sigma_rule_file": null,
"sigma_rule_id": null,
"stored_query_id": null,
"query_dsl": null,
"query_name": "gulpfilter-ce9da5bb-9309-4a6e-9724-86b959ba311d"
}
],
"live": false,
"matched_events_total": 9037,
"ingest_errors": null,
"current_src_file": null
},
"reqId": "639432bd-db7f-4a7f-a226-400a77af4c92",
"timestamp": "2024-07-04T20:24:56.006Z"
}

The basic data for each event allows to correctly insert it into the right container (context name, source name) and draw it (timestamp and event code). Note: The naming here is a bit different because we’re adhering to ECS mapping for compatibility: context=source.domain and sourcefile=log.file.path

Paging results

GulpQueryFilter has a default limit of 1000 results per query (the amount can be changed in the request), when more results are available, the “search_after” property is populated with the next timestamp and event id to be used for the next request (i.e. to retrieve the next “page” of results – just use the same search_after in the response). The fetching loop ends when the “search_after” object in the response is null, that is, when there are no more results.

Drawing events

Events are drawn based on their basic properties: @timestamp, event.duration and event.code. The first two allow to “place” the event on the timeline, while the latter determines the event’s color.

Thanks to the local max and min event codes for each source, we can apply a linear interpolation to map the event code to a color in a fixed-size gradient array. At the moment the only used palette is FLIR’s “ironbow”, and some others have been set up (matplotlib’s Viridis, Stragma, etc). In the future, users should be able to configure each source’s palette, as well as use their own gradients.

Depending on the type of the event, that is the type of the sourcefile, a different render can be used. Currently 3 type specific renderers are available, namely win_evtx (windows evtx events) apache_clf (common log format) and a generic event renderer (similar to win_evtx with no gradient). Two more example generic purpose renderers are histogram and plot, whose main function is to give an idea of the density of events across time chunks. The actual implementation is just demonstrative and should be reviewed.

‍[!NOTE] The actual desktop client can be extended with additional custom rendering plugins, and we were imagining a custom simple DSL in the future to allow non programmers making their own rendering plugins. Take this into consideration when designing the code architecture of the web client (the DSL could be overkill, maybe just a simple js interface with drawing primitives?)

14.png

Notes, links and other collaboration objects

Similarly to events, the following collabobjects can be retrieved: Notes → main properties are: title, description, glyph (icon) and a timepin (if a note is not related to any event) or a list of related events. When a note has multiple related events, it should be shown along each event it refers to. A color attribute is also available, originally used to dye icon as a further mean of color coding notes (up to the). /note_list Links → main properties are: one source event, one or more target events /link_list Highligths → …

‍[!NOTE] collabobj apis should be adapted to match the event api behaviour, adding the possibility of querying a defined time range and retrieving a minimal subset of information.

Event / note details

When clicking on an event or note, the corresponding details are shown in the rightmost area of the screen. Event details in the upper box, note details in the lower.

TODO: -let the user switch between the raw text and the json tree representation

Api: /query_single_event ( possibily /query_single_note when it will be added)

15.png

Using the main view

This view can be panned (moved) by left clicking and dragging the mouse around, and zoomed using the mouse wheel or right clicking to draw a zoom area. A sort of magnifying glass to aid clicking on thin events can be brought up by pressing the shift key while the mouse cursor is in the main event area.

16.png

It is also possible to quickly go to the next or previous event, relative to the currently selected one, by pressing the A and D keys, respectively.

Finally, a large menu bar at the bottom hosts buttons related to:

Session menu

17.png

  • Clear session → clears everything (not sure it’s currently useful, it was in the legacy stand-alone version)
  • Load session → load a previously saved session (working in the legacy version, needs to be converted using the web api, using the “session” collabobject)
  • Save session → same as above
  • Show/hide links
  • Swap light/dark theme
  • Upload file → allows uploading new files to be ingested by the backend or sigma rules to be applied to the events.
  • Save current screen – exports the currently shown screen to an image
  • Export to svg – exports the currently shown sources to an svg file, if supported by the source type renderer. Most svg rendering functions are not up to date with the screen rendering counterparts.
  • Debug menu - shows a debug menu (internal use only)
  • Refresh data – reloads all data from the server

Timeview menu

18.png

  • “Zoom fit” → zooms out as far as possible to fit all events (in the current session) into the window
  • Select new range → shows the range selection window (just like at the beginning of the session)
  • Zoom → allows to zoom on a specified area (same as right-clicking)
  • Go to time → ask the user for a timestamp and center it on the screen
  • Add / delete row → manually add or delete a source track (not used anymore – can be removed?)
  • Highlight → draws an highlighted area (this is not used at the moment, and should be mapped to a collab object named “highlight”)
  • Manually draw link → this feature is not used anymore, links should be created through the event/note menu.

Event/note menu:

19.png

  • Add / edit note → Adds a new note to the selected event, or edit an existing one
  • Go to event → when a note is clicked, go to the corresponding event (This is now deprecated, as both the event and the note are shown, vertically stacked, in the leftmost area)
  • Delete note → no explanation needed
  • Delete event → manually delete an nevent (not used anymore -should be removed?)
  • Set link start → marks the currently selected event as the start of a link
  • Set link end → marks the currently selected event as the end of a link (when both the start and end points are selected, a new link is created)
  • Show note list – shows the list of available notes.

Adding a note

20.png

Api /note_create

Uploading a file

21.png

The file upload window allow the user to upload new files to be ingested or rules to be processed. Currently the file format is lamely guessed by its extension.

Api /ingest_file

TODO:

  • Format identification should be thoroughly implemented on the client or server side, as it impacts on how the file will be ingested (i.e. the appropriate ingestion plugin that should be used). Users should be able to override the ingestion plugin used regardless of the identified format.
  • The user should specify the context for which the file is being ingested, and possibly be able to override the name! If a file .zip is provided and contains a file named “manifest.json” in the root of the archive, it should be treated as a special “format” and the included manifest file should be used to identify the format of each file in the archive instead.