Inheritance diagram for PluginBase:

Collaboration diagram for PluginBase:

Public Member Functions
None	__init__ (self, str path, **kwargs)

None	cleanup (self)

list[str]	depends_on (self)

str	desc (self)

str	event_type_field (self)

str	get_unmapped_field_name (self, str field)

GulpRequestStatus	ingest (self, str index, str req_id, int client_id, int operation_id, str context, str\|list[dict] source, str ws_id, GulpPluginParams plugin_params=None, GulpIngestionFilter flt=None, **kwargs)

tuple[dict, GulpMapping]	ingest_plugin_initialize (self, str index, str\|dict source, bool skip_mapping=False, ProcessingPipeline pipeline=None, str mapping_file=None, str mapping_id=None, GulpPluginParams plugin_params=None)

bool	internal (self)

str	name (self)

list[GulpPluginOption]	options (self)

ProcessingPipeline	pipeline (self, GulpPluginParams plugin_params=None, **kwargs)

tuple[int, GulpRequestStatus]	query (self, int operation_id, int client_id, int user_id, str username, str ws_id, str req_id, GulpPluginParams plugin_params, GulpQueryFilter flt, GulpQueryOptions options=None)

list[GulpDocument]	record_to_gulp_document (self, int operation_id, int client_id, str context, str source, TmpIngestStats fs, any record, int record_idx, GulpMapping custom_mapping=None, dict index_type_mapping=None, str plugin=None, GulpPluginParams plugin_params=None, **kwargs)

ProcessingPipeline	sigma_plugin_initialize (self, ProcessingPipeline pipeline=None, str mapping_file=None, str mapping_id=None, str product=None, GulpPluginParams plugin_params=None)

list[str]	tags (self)

GulpPluginType	type (self)

str	version (self)

Public Attributes
list	buffer = []

str	client_id = None

str	context = None

str	index = None

str	operation_id = None

	path = path

str	req_id = None

str	ws_id = None

Protected Member Functions
list[GulpDocument]	_build_gulpdocuments (self, list[FieldMappingEntry] fme, int idx, int operation_id, str context, str plugin, int client_id, str raw_event, str original_id, str src_file, int timestamp=None, int timestamp_nsec=None, str event_code=None, list[str] cat=None, int duration_nsec=0, GulpLogLevel gulp_log_level=None, str original_log_level=None, bool remove_raw_event=False, **kwargs)

list[dict]	_build_ingestion_chunk_for_ws (self, list[dict] docs, GulpIngestionFilter flt=None)

list[GulpDocument]	_call_record_to_gulp_document_funcs (self, int operation_id, int client_id, str context, str source, TmpIngestStats fs, any record, int record_idx, GulpMapping custom_mapping=None, dict index_type_mapping=None, str plugin=None, GulpPluginParams plugin_params=None, Callable record_to_gulp_document_fun=None, **kwargs)

GulpRequestStatus	_finish_ingestion (self, str index, str\|dict source, str req_id, int client_id, str ws_id, TmpIngestStats fs, GulpIngestionFilter flt=None)

TmpIngestStats	_flush_buffer (self, str index, TmpIngestStats fs, str ws_id, str req_id, GulpIngestionFilter flt=None, bool wait_for_refresh=False)

TmpIngestStats	_ingest_record (self, str index, GulpDocument\|dict doc, TmpIngestStats fs, str ws_id, str req_id, GulpIngestionFilter flt=None, bool flush_enabled=True, **kwargs)

list[FieldMappingEntry]	_map_source_key (self, GulpPluginParams plugin_params, GulpMapping custom_mapping, str source_key, Any v, dict index_type_mapping=None, bool ignore_custom_mapping=False, **kwargs)

TmpIngestStats	_parser_failed (self, TmpIngestStats fs, str\|dict source, Exception\|str ex)

tuple[GulpMapping, GulpPluginParams]	_process_plugin_params (self, GulpMapping custom_mapping, GulpPluginParams plugin_params=None)

tuple[TmpIngestStats, bool]	_process_record (self, str index, any record, int record_idx, Callable my_record_to_gulp_document_fun, str ws_id, str req_id, int operation_id, int client_id, str context, str source, TmpIngestStats fs, GulpMapping custom_mapping=None, dict index_type_mapping=None, str plugin=None, GulpPluginParams plugin_params=None, GulpIngestionFilter flt=None, **kwargs)

TmpIngestStats	_record_failed (self, TmpIngestStats fs, any entry, str\|dict source, Exception\|str ex)

any	_type_checks (self, any v, str k, dict index_type_mapping)

Detailed Description

Base class for all Gulp plugins.

Definition at line 46 of file plugin.py.

Constructor & Destructor Documentation

◆ init()

None __init__	(		self,
		str	path,
		**	kwargs )

Initializes a new instance of the PluginBase class.

Args:
    path (str): The path to the plugin.
    kwargs: additional arguments if any

Definition at line 51 of file plugin.py.

    ) -> None:
        """
        Initializes a new instance of the PluginBase class.
 
        Args:
            path (str): The path to the plugin.
            kwargs: additional arguments if any
        """
        self.req_id: str = None
        self.index: str = None
        self.client_id: str = None
        self.operation_id: str = None
        self.context: str = None
        self.ws_id: str = None
        self.path = path
 
        self.buffer: list[GulpDocument] = []
        for k, v in kwargs.items():
            self.__dict__[k] = v
 
        super().__init__()
 

Member Function Documentation

◆ _build_gulpdocuments()

list[GulpDocument] _build_gulpdocuments	(		self,
		list[FieldMappingEntry]	fme,
		int	idx,
		int	operation_id,
		str	context,
		str	plugin,
		int	client_id,
		str	raw_event,
		str	original_id,
		str	src_file,
		int	timestamp = None,
		int	timestamp_nsec = None,
		str	event_code = None,
		list[str]	cat = None,
		int	duration_nsec = 0,
		GulpLogLevel	gulp_log_level = None,
		str	original_log_level = None,
		bool	remove_raw_event = False,
		**	kwargs )

protected

build one or more GulpDocument objects from a list of FieldMappingEntry objects:

this function creates as many GulpDocument objects as there are FieldMappingEntry objects with is_timestamp=True.
if no FieldMappingEntry object has is_timestamp=True, it creates a single GulpDocument object with the first FieldMappingEntry object.

Definition at line 558 of file plugin.py.

    ) -> list[GulpDocument]:
        """
        build one or more GulpDocument objects from a list of FieldMappingEntry objects:
 
        this function creates as many GulpDocument objects as there are FieldMappingEntry objects with is_timestamp=True.
        if no FieldMappingEntry object has is_timestamp=True, it creates a single GulpDocument object with the first FieldMappingEntry object.
        """
        docs: list[GulpDocument] = []
        append_doc = docs.append  # local variable for faster access
 
        common_params = {
            "idx": idx,
            "operation_id": operation_id,
            "context": context,
            "plugin": plugin,
            "client_id": client_id,
            "raw_event": raw_event,
            "original_id": original_id,
            "src_file": src_file,
            "timestamp": timestamp,
            "timestamp_nsec": timestamp_nsec,
            "event_code": event_code,
            "cat": cat,
            "duration_nsec": duration_nsec,
            "gulp_log_level": gulp_log_level,
            "original_log_level": original_log_level,
            **kwargs,
        }
        for f in fme:
            # print("%s\n\n" % (f))
            # for each is_timestamp build a gulpdocument with all the fields in fme
            if f.is_timestamp:
                d = GulpDocument(fme=fme, f=f, **common_params)
                if remove_raw_event:
                    d.original_event = None
 
                # print("%s\n\n" % (d))
                append_doc(d)
 
        if len(docs) == 0:
            # create a document with the given timestamp in timestamp/timestamp_nsec (if any, either it will be set to 0/invalid)
            d = GulpDocument(fme=fme, **common_params)
            if remove_raw_event:
                d.original_event = None
            append_doc(d)
 
        return docs
 

◆ _build_ingestion_chunk_for_ws()

list[dict] _build_ingestion_chunk_for_ws	(		self,
		list[dict]	docs,
		GulpIngestionFilter	flt = None )

protected

Builds the ingestion chunk for the websocket, filtering if needed.

Definition at line 829 of file plugin.py.

    ) -> list[dict]:
        """
        Builds the ingestion chunk for the websocket, filtering if needed.
        """
        # logger().debug("building ingestion chunk, flt=%s" % (flt))
        if not docs:
            return []
 
        ws_docs = [
            {
                "_id": doc["_id"],
                "@timestamp": doc["@timestamp"],
                "gulp.source.file": doc["gulp.source.file"],
                "event.duration": doc["event.duration"],
                "gulp.context": doc["gulp.context"],
                "gulp.log.level": doc.get("gulp.log.level", int(GulpLogLevel.INFO)),
                "event.category": doc.get("event.category", None),
                "event.code": doc["event.code"],
                "gulp.event.code": doc["gulp.event.code"],
            }
            for doc in docs
            if elastic_api.filter_doc_for_ingestion(
                doc, flt, ignore_store_all_documents=True
            )
            == GulpEventFilterResult.ACCEPT
        ]
 
        return ws_docs
 

Here is the caller graph for this function:

◆ _call_record_to_gulp_document_funcs()

list[GulpDocument] _call_record_to_gulp_document_funcs	(		self,
		int	operation_id,
		int	client_id,
		str	context,
		str	source,
		TmpIngestStats	fs,
		any	record,
		int	record_idx,
		GulpMapping	custom_mapping = None,
		dict	index_type_mapping = None,
		str	plugin = None,
		GulpPluginParams	plugin_params = None,
		Callable	record_to_gulp_document_fun = None,
		**	kwargs )

protected

Stub function to call stacked plugins record_to_document_gulp_document.
Each function is called with the previously returned GulpDocument.

Args:
    operation_id (int): the operation ID associated with the record
    client_id (int): client ID performing the ingestion
    context (str): context associated with the record
    source (str): source of the record (source file name or path, usually)
    fs (TmpIngestStats): _description_
    record (any): a single record (first time) or a list of GulpDocument objects (in stacked plugins)
    record_idx (int): The index of the record in source.
    custom_mapping (GulpMapping, optional): The custom mapping to use for the conversion. Defaults to None.
    index_type_mapping (dict, optional): elastic search index type mappings { "field": "type", ... }. Defaults to None.
    plugin (str, optional): "agent.type" to be set in the GulpDocument. Defaults to None.
    plugin_params (GulpPluginParams, optional): The plugin parameters to use, if any. Defaults to None.
    record_to_gulp_document_fun (Callable, optional): function to parse record into a gulp document, if stacked this receives a list of GulpDocuments

Returns:
    list[GulpDocument]: zero or more GulpDocument objects

Definition at line 379 of file plugin.py.

    ) -> list[GulpDocument]:
        """Stub function to call stacked plugins record_to_document_gulp_document.
        Each function is called with the previously returned GulpDocument.
 
        Args:
            operation_id (int): the operation ID associated with the record
            client_id (int): client ID performing the ingestion
            context (str): context associated with the record
            source (str): source of the record (source file name or path, usually)
            fs (TmpIngestStats): _description_
            record (any): a single record (first time) or a list of GulpDocument objects (in stacked plugins)
            record_idx (int): The index of the record in source.
            custom_mapping (GulpMapping, optional): The custom mapping to use for the conversion. Defaults to None.
            index_type_mapping (dict, optional): elastic search index type mappings { "field": "type", ... }. Defaults to None.
            plugin (str, optional): "agent.type" to be set in the GulpDocument. Defaults to None.
            plugin_params (GulpPluginParams, optional): The plugin parameters to use, if any. Defaults to None.
            record_to_gulp_document_fun (Callable, optional): function to parse record into a gulp document, if stacked this receives a list of GulpDocuments
 
        Returns:
            list[GulpDocument]: zero or more GulpDocument objects
        """
        # plugin_params=deepcopy(plugin_params)
 
        if plugin_params is None:
            plugin_params = GulpPluginParams()
 
        docs = record
 
        if record_to_gulp_document_fun is not None:
            docs = await record_to_gulp_document_fun(
                operation_id,
                client_id,
                context,
                source,
                fs,
                record,
                record_idx,
                custom_mapping,
                index_type_mapping,
                plugin,
                plugin_params,
                **kwargs,
            )
 
        for fun in plugin_params.record_to_gulp_document_fun:
            docs = await fun(
                operation_id,
                client_id,
                context,
                source,
                fs,
                docs,
                record_idx,
                custom_mapping,
                index_type_mapping,
                plugin,
                plugin_params,
                **kwargs,
            )
 
        if docs is None:
            return []
        return docs
 

Here is the caller graph for this function:

◆ _finish_ingestion()

GulpRequestStatus _finish_ingestion	(		self,
		str	index,
		str \| dict	source,
		str	req_id,
		int	client_id,
		str	ws_id,
		TmpIngestStats	fs,
		GulpIngestionFilter	flt = None )

protected

to be called whenever ingest() must exit: flushes the buffer and updates the ingestion stats

Definition at line 970 of file plugin.py.

    ) -> GulpRequestStatus:
        """
        to be called whenever ingest() must exit: flushes the buffer and updates the ingestion stats
        """
        try:
            # finally flush ingestion buffer
            fs = await self._flush_buffer(index, fs, ws_id, req_id, flt)
            logger().info(
                "INGESTION DONE FOR source=%s,\n\tclient_id=%d (processed(ingested)=%d, failed=%d, skipped=%d, errors=%d, parser_errors=%d)"
                % (
                    muty.string.make_shorter(str(source), 260),
                    client_id,
                    fs.ev_processed,
                    fs.ev_failed,
                    fs.ev_skipped,
                    len(fs.ingest_errors),
                    fs.parser_failed,
                )
            )
        except Exception as ex:
            fs = fs.update(ingest_errors=[ex])
            logger().exception(
                "FAILED finalizing ingestion for source=%s"
                % (muty.string.make_shorter(str(source), 260))
            )
 
        finally:
            status, _ = await GulpStats.update(
                await collab_api.collab(),
                req_id,
                ws_id,
                fs=fs,
                force=True,
                file_done=True,
            )
        return status
 
 

Here is the call graph for this function:

◆ _flush_buffer()

TmpIngestStats _flush_buffer	(		self,
		str	index,
		TmpIngestStats	fs,
		str	ws_id,
		str	req_id,
		GulpIngestionFilter	flt = None,
		bool	wait_for_refresh = False )

protected

NOTE: errors appended by this function are intended as INGESTION errors:
it means something wrong with the format of the event, and must be fixed ASAP if this happens.
ideally, function should NEVER append errors and the errors total should be the same before and after this function returns (this function may only change the skipped total, which means some duplicates were found).

Definition at line 860 of file plugin.py.

    ) -> TmpIngestStats:
        """
        NOTE: errors appended by this function are intended as INGESTION errors:
        it means something wrong with the format of the event, and must be fixed ASAP if this happens.
        ideally, function should NEVER append errors and the errors total should be the same before and after this function returns (this function may only change the skipped total, which means some duplicates were found).
        """
        if len(self.buffer) == 0:
            # already flushed
            return fs
 
        # logger().debug('flushing ingestion buffer, len=%d' % (len(self.buffer)))
 
        skipped, failed, failed_ar, ingested_docs = await elastic_api.ingest_bulk(
            elastic_api.elastic(),
            index,
            self.buffer,
            flt=flt,
            wait_for_refresh=wait_for_refresh,
        )
        # print(json.dumps(ingested_docs, indent=2))
 
        if failed > 0:
            if config.debug_abort_on_elasticsearch_ingestion_error():
                raise Exception(
                    "elasticsearch ingestion errors means GulpDocument contains invalid data, review errors on collab db!"
                )
 
        self.buffer = []
 
        # build ingestion chunk
        ws_docs = self._build_ingestion_chunk_for_ws(ingested_docs, flt)
 
        # send ingested docs to websocket
        if len(ws_docs) > 0:
            ws_api.shared_queue_add_data(
                WsQueueDataType.INGESTION_CHUNK,
                req_id,
                {"plugin": self.name(), "events": ws_docs},
                ws_id=ws_id,
            )
 
        # update stats
        fs = fs.update(
            failed=failed,
            skipped=skipped,
            ingest_errors=failed_ar,
        )
        return fs
 

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _ingest_record()

TmpIngestStats _ingest_record	(		self,
		str	index,
		GulpDocument \| dict	doc,
		TmpIngestStats	fs,
		str	ws_id,
		str	req_id,
		GulpIngestionFilter	flt = None,
		bool	flush_enabled = True,
		**	kwargs )

protected

bufferize as much as ingestion_buffer_size, then flush (writes to elasticsearch)

Definition at line 917 of file plugin.py.

    ) -> TmpIngestStats:
        """
        bufferize as much as ingestion_buffer_size, then flush (writes to elasticsearch)
        """
        ingestion_buffer_size = config.config().get("ingestion_buffer_size", 1000)
        self.buffer.append(doc)
        if len(self.buffer) >= ingestion_buffer_size and flush_enabled:
            # time to flush
            fs = await self._flush_buffer(index, fs, ws_id, req_id, flt)
 
        return fs
 

Here is the call graph for this function:

Here is the caller graph for this function:

◆ _map_source_key()

list[FieldMappingEntry] _map_source_key	(		self,
		GulpPluginParams	plugin_params,
		GulpMapping	custom_mapping,
		str	source_key,
		Any	v,
		dict	index_type_mapping = None,
		bool	ignore_custom_mapping = False,
		**	kwargs )

protected

map source key to a field mapping entry with "result": {mapped_key: v}

Args:
    plugin_params (GulpPluginParams): The plugin parameters.
    custom_mapping (GulpMapping): The custom mapping.
    source_key (str): The key to look for(=the event record key to be mapped) in the custom_mapping dictionary
    v (any): value to set for mapped key/s.
    index_type_mapping (dict, optional): The elasticsearch index key->type mappings. Defaults to None.
    ignore_custom_mapping (bool, optional): Whether to ignore custom_mapping and directly map source_key to v. Defaults to False.
    kwargs: Additional keyword arguments.

Returns:
    list[FieldMappingEntry]: zero or more FieldMappingEntry objects with "result" set.

Definition at line 686 of file plugin.py.

    ) -> list[FieldMappingEntry]:
        """
        map source key to a field mapping entry with "result": {mapped_key: v}
 
        Args:
            plugin_params (GulpPluginParams): The plugin parameters.
            custom_mapping (GulpMapping): The custom mapping.
            source_key (str): The key to look for(=the event record key to be mapped) in the custom_mapping dictionary
            v (any): value to set for mapped key/s.
            index_type_mapping (dict, optional): The elasticsearch index key->type mappings. Defaults to None.
            ignore_custom_mapping (bool, optional): Whether to ignore custom_mapping and directly map source_key to v. Defaults to False.
            kwargs: Additional keyword arguments.
 
        Returns:
            list[FieldMappingEntry]: zero or more FieldMappingEntry objects with "result" set.
        """
        # get mapping and option from custom_mapping
        if index_type_mapping is None:
            index_type_mapping = {}
        # logger().debug('len index type mapping=%d' % (len(index_type_mapping)))
        mapping_dict: dict = custom_mapping.fields
        mapping_options = (
            custom_mapping.options
            if custom_mapping.options is not None
            else GulpMappingOptions()
        )
 
        # basic checks
        if v == "-" or v is None:
            return []
 
        if isinstance(v, str):
            v = v.strip()
            if not v and mapping_options.ignore_blanks:
                # not adding blank strings
                return []
 
        # fix value if needed, and add to extra
        if ignore_custom_mapping or (
            plugin_params is not None and plugin_params.ignore_mapping_ingest
        ):
            # direct mapping, no need to check custom_mappings
            return [FieldMappingEntry(result={source_key: v})]
 
        if source_key not in mapping_dict:
            # logger().error('key "%s" not found in custom mapping, mapping_dict=%s!' % (source_key, muty.string.make_shorter(str(mapping_dict))))
            # key not found in custom_mapping, check if we have to map it anyway
            if not mapping_options.ignore_unmapped:
                return [
                    FieldMappingEntry(
                        result={self.get_unmapped_field_name(source_key): str(v)}
                    )
                ]
 
        # there is a mapping defined to be processed
        fm: FieldMappingEntry = mapping_dict[source_key]
        map_to_list = (
            [fm.map_to] if isinstance(fm.map_to, (str, type(None))) else fm.map_to
        )
 
        # in the end, this function will return a list of FieldMappingEntry objects with "result" set: these results will be used to create the GulpDocument object
        fme_list: list[FieldMappingEntry] = []
        for k in map_to_list:
            # make a copy of fme without using deepcopy)
            dest_fm = FieldMappingEntry(
                is_timestamp=fm.is_timestamp,
                event_code=fm.event_code,
                do_multiply=fm.do_multiply,
                is_timestamp_chrome=fm.is_timestamp_chrome,
                is_variable_mapping=fm.is_variable_mapping,
                result={},
            )
 
            # check if it is a number and/or a timestamp (including chrome timestamp, which is a special case)
            is_numeric = isinstance(v, int) or str(v).isnumeric()
            if is_numeric:
                v = int(v)
                # ensure chrome timestamp is properly converted to nanos
                # logger().debug('***** is_numeric, v=%d' % (v))
                if fm.is_timestamp_chrome:
                    v = int(muty.time.chrome_epoch_to_nanos(v))
                    # logger().debug('***** is_timestamp_chrome, v nsec=%d' % (v))
 
                if fm.do_multiply is not None:
                    # apply a multipler if any (must turn v to nanoseconds)
                    # logger().debug("***** is_numeric, multiply, v=%d" % (v))
                    v = int(v * fm.do_multiply)
                    # logger().debug("***** is_numeric, AFTER multiply, v=%d" % (v))
 
            elif isinstance(v, str) and fm.is_timestamp:
                v = int(
                    muty.time.string_to_epoch_nsec(
                        v,
                        utc=mapping_options.timestamp_utc,
                        dayfirst=mapping_options.timestamp_dayfirst,
                        yearfirst=mapping_options.timestamp_yearfirst,
                    )
                )
                # logger().debug('***** str and is_timestamp, v nsec=%d' % (v))
            if fm.is_timestamp:
                # it's a timestamp, another event will be generated
                vv = muty.time.nanos_to_millis(v)
                dest_fm.result["@timestamp"] = vv
                dest_fm.result["@timestamp_nsec"] = v
                # logger().debug('***** timestamp nanos, v=%d' % (v))
                # logger().debug('***** timestamp to millis, v=%d' % (vv))
 
            if fm.is_timestamp or fm.is_timestamp_chrome:
                # logger().debug('***** timestamp or timestamp_chrome, v=%d' % (v))
                if v < 0:
                    # logger().debug('***** adding invalid timestamp')
                    v = 0
                    GulpDocument.add_invalid_timestamp(dest_fm.result)
                if k is not None:
                    # also add to mapped key
                    dest_fm.result[k] = v
            else:
                # not a timestamp, map
                if k is None:
                    # add unmapped
                    k = self.get_unmapped_field_name(source_key)
                else:
                    v = self._type_checks(v, k, index_type_mapping)
                dest_fm.result[k] = v
 
            fme_list.append(dest_fm)
            """
            logger().debug('FME LIST FOR THIS RECORD:')
            for p in fme_list:
                logger().debug(p)
            logger().debug('---------------------------------')
            """
        return fme_list
 

Here is the call graph for this function:

◆ _parser_failed()

TmpIngestStats _parser_failed	(		self,
		TmpIngestStats	fs,
		str \| dict	source,
		Exception \| str	ex )

protected

whole source failed ingestion (error happened before the record parsing loop), helper to update stats

Definition at line 939 of file plugin.py.

    ) -> TmpIngestStats:
        """
        whole source failed ingestion (error happened before the record parsing loop), helper to update stats
        """
        logger().exception(
            "PARSER FAILED: source=%s, ex=%s"
            % (muty.string.make_shorter(str(source), 260), ex)
        )
        fs = fs.update(ingest_errors=[ex], parser_failed=1)
        return fs
 

◆ _process_plugin_params()

tuple[GulpMapping, GulpPluginParams] _process_plugin_params	(		self,
		GulpMapping	custom_mapping,
		GulpPluginParams	plugin_params = None )

protected

Process the plugin parameters checking parameters as `timestamp_field`, ... and update the custom mapping accordingly.

Args:
    custom_mapping (GulpMapping): The custom mapping provided: if it is not empty, it will be used as is (plugin_params will be ignored)
    plugin_params (GulpPluginParams, optional): The plugin parameters. Defaults to None.

Returns:
    tuple[GulpMapping, GulpPluginParams]: The updated custom mapping and plugin parameters.

Definition at line 169 of file plugin.py.

    ) -> tuple[GulpMapping, GulpPluginParams]:
        """
        Process the plugin parameters checking parameters as `timestamp_field`, ... and update the custom mapping accordingly.
 
        Args:
            custom_mapping (GulpMapping): The custom mapping provided: if it is not empty, it will be used as is (plugin_params will be ignored)
            plugin_params (GulpPluginParams, optional): The plugin parameters. Defaults to None.
 
        Returns:
            tuple[GulpMapping, GulpPluginParams]: The updated custom mapping and plugin parameters.
        """
        if plugin_params is None:
            plugin_params = GulpPluginParams()
 
        if len(custom_mapping.fields) > 0:
            # custom_mapping provided
            return custom_mapping, plugin_params
 
        logger().warning("no custom mapping provided")
 
        tf = plugin_params.timestamp_field
        if tf is not None:
            logger().debug("using timestamp_field=%s" % (tf))
            # build a proper custom_mapping with just timestamp
            custom_mapping.fields[tf] = FieldMappingEntry(is_timestamp=True)
        # logger().debug(custom_mapping)
        return custom_mapping, plugin_params
 

◆ _process_record()

tuple[TmpIngestStats, bool] _process_record	(		self,
		str	index,
		any	record,
		int	record_idx,
		Callable	my_record_to_gulp_document_fun,
		str	ws_id,
		str	req_id,
		int	operation_id,
		int	client_id,
		str	context,
		str	source,
		TmpIngestStats	fs,
		GulpMapping	custom_mapping = None,
		dict	index_type_mapping = None,
		str	plugin = None,
		GulpPluginParams	plugin_params = None,
		GulpIngestionFilter	flt = None,
		**	kwargs )

protected

Process a record for ingestion, updating ingestion stats.
Args:
    index (str): The index to ingest the record into.
    record (any): The record to process.
    record_idx (int): The index of the record as in the source.
    my_record_to_gulp_document_fun (Callable): The function (for this plugin) to convert the record to one or more GulpDocument/s.
    ws_id (str): The websocket ID
    req_id (str): The request ID.
    operation_id (int): The operation ID.
    client_id (int): The client ID.
    context (str): The context of the record.
    source (str): The source of the record.
    fs (TmpIngestStats): The temporary ingestion statistics.
    custom_mapping (GulpMapping, optional): The custom mapping for the record. Defaults to None.
    index_type_mapping (dict, optional): The elastic index type mapping. Defaults to None.
    plugin (str, optional): The plugin name. Defaults to None.
    plugin_params (GulpPluginParams, optional): The plugin parameters. Defaults to None.
    flt (GulpIngestionFilter, optional): The ingestion filter. Defaults to None.
    **kwargs: Additional keyword arguments.
Returns:
    tuple[TmpIngestStats, bool]: A tuple containing the updated temporary ingestion statistics and a flag indicating whether to break the ingestion process.

Definition at line 243 of file plugin.py.

    ) -> tuple[TmpIngestStats, bool]:
        """
        Process a record for ingestion, updating ingestion stats.
        Args:
            index (str): The index to ingest the record into.
            record (any): The record to process.
            record_idx (int): The index of the record as in the source.
            my_record_to_gulp_document_fun (Callable): The function (for this plugin) to convert the record to one or more GulpDocument/s.
            ws_id (str): The websocket ID
            req_id (str): The request ID.
            operation_id (int): The operation ID.
            client_id (int): The client ID.
            context (str): The context of the record.
            source (str): The source of the record.
            fs (TmpIngestStats): The temporary ingestion statistics.
            custom_mapping (GulpMapping, optional): The custom mapping for the record. Defaults to None.
            index_type_mapping (dict, optional): The elastic index type mapping. Defaults to None.
            plugin (str, optional): The plugin name. Defaults to None.
            plugin_params (GulpPluginParams, optional): The plugin parameters. Defaults to None.
            flt (GulpIngestionFilter, optional): The ingestion filter. Defaults to None.
            **kwargs: Additional keyword arguments.
        Returns:
            tuple[TmpIngestStats, bool]: A tuple containing the updated temporary ingestion statistics and a flag indicating whether to break the ingestion process.
        """
 
        # convert record to one or more GulpDocument objects
        docs = await self._call_record_to_gulp_document_funcs(
            operation_id=operation_id,
            client_id=client_id,
            context=context,
            source=source,
            fs=fs,
            record=record,
            record_idx=record_idx,
            custom_mapping=custom_mapping,
            index_type_mapping=index_type_mapping,
            plugin=plugin,
            plugin_params=plugin_params,
            record_to_gulp_document_fun=my_record_to_gulp_document_fun,
            **kwargs,
        )
        # ingest record
        for d in docs:
            fs = await self._ingest_record(index, d, fs, ws_id, req_id, flt, **kwargs)
 
        status, _ = await GulpStats.update(
            await collab_api.collab(),
            req_id,
            ws_id,
            fs=fs.update(processed=len(docs)),
        )
        must_break = False
        if status in [GulpRequestStatus.FAILED, GulpRequestStatus.CANCELED]:
            must_break = True
 
        return fs, must_break
 

Here is the call graph for this function:

◆ _record_failed()

TmpIngestStats _record_failed	(		self,
		TmpIngestStats	fs,
		any	entry,
		str \| dict	source,
		Exception \| str	ex )

protected

record failed ingestion (in the record parser loop), helper to update stats

Args:
    fs (TmpIngestStats): The temporary ingestion statistics.
    entry (any): The entry that failed.
    source (str): The source of the record.
    ex (Exception|str): The exception that caused the failure.
Returns:
    TmpIngestStats: The updated temporary ingestion statistics.

Definition at line 952 of file plugin.py.

    ) -> TmpIngestStats:
        """
        record failed ingestion (in the record parser loop), helper to update stats
 
        Args:
            fs (TmpIngestStats): The temporary ingestion statistics.
            entry (any): The entry that failed.
            source (str): The source of the record.
            ex (Exception|str): The exception that caused the failure.
        Returns:
            TmpIngestStats: The updated temporary ingestion statistics.
        """
        # logger().exception("RECORD FAILED: source=%s, record=%s, ex=%s" % (muty.string.make_shorter(str(source),260), muty.string.make_shorter(str(entry),260), ex))
        fs = fs.update(failed=1, ingest_errors=[ex])
        return fs
 

◆ _type_checks()

any _type_checks	(		self,
		any	v,
		str	k,
		dict	index_type_mapping )

protected

check if the value should be fixed based on the index type mapping

Args:
    v (any): The value to check.
    k (str): The mapped field (i.e. "user.id", may also be an unmapped (i.e. "gulp.unmapped") field)
    index_type_mapping (dict): The elasticsearch index key->type mappings.

Definition at line 641 of file plugin.py.

    def _type_checks(self, v: any, k: str, index_type_mapping: dict) -> any:
        """
        check if the value should be fixed based on the index type mapping
 
        Args:
            v (any): The value to check.
            k (str): The mapped field (i.e. "user.id", may also be an unmapped (i.e. "gulp.unmapped") field)
            index_type_mapping (dict): The elasticsearch index key->type mappings.
        """
        if k not in index_type_mapping:
            # logger().debug("key %s not found in index_type_mapping" % (k))
            return str(v)
 
        index_type = index_type_mapping[k]
        if index_type == "long":
            # logger().debug("converting %s:%s to long" % (k, v))
            if isinstance(v, str):
                if v.isnumeric():
                    return int(v)
                if v.startswith("0x"):
                    return int(v, 16)
            return v
 
        if index_type == "date" and isinstance(v, str) and v.lower().startswith("0x"):
            # convert hex to int
            return int(v, 16)
 
        if index_type == "keyword" or index_type == "text":
            # logger().debug("converting %s:%s to keyword" % (k, v))
            return str(v)
 
        if index_type == "ip":
            # logger().debug("converting %s:%s to ip" % (k, v))
            if "local" in v.lower():
                return "127.0.0.1"
            try:
                ipaddress.ip_address(v)
            except ValueError as ex:
                logger().exception(ex)
                return None
 
        # add more types here if needed ...
        # logger().debug("returning %s:%s" % (k, v))
        return str(v)
 

Here is the caller graph for this function:

◆ cleanup()

None cleanup ( self )

Optional cleanup routine to call on unload.

Definition at line 552 of file plugin.py.

    def cleanup(self) -> None:
        """
        Optional cleanup routine to call on unload.
        """
        return
 

◆ depends_on()

list[str] depends_on ( self )

Returns a list of plugins this plugin depends on.

Definition at line 113 of file plugin.py.

    def depends_on(self) -> list[str]:
        """
        Returns a list of plugins this plugin depends on.
        """
        return []
 

◆ desc()

str desc ( self )

Returns a description of the plugin.

Definition at line 96 of file plugin.py.

    def desc(self) -> str:
        """
        Returns a description of the plugin.
        """
 

Here is the caller graph for this function:

◆ event_type_field()

str event_type_field ( self )

Returns the field name for the event type.

Definition at line 107 of file plugin.py.

    def event_type_field(self) -> str:
        """
        Returns the field name for the event type.
        """
        return "event.code"
 

◆ get_unmapped_field_name()

str get_unmapped_field_name	(		self,
		str	field )

Returns the name of the unmapped field.

Parameters:
- field (str): The name of the field.

Returns:
- str: The name of the unmapped field.

Definition at line 626 of file plugin.py.

    def get_unmapped_field_name(self, field: str) -> str:
        """
        Returns the name of the unmapped field.
 
        Parameters:
        - field (str): The name of the field.
 
        Returns:
        - str: The name of the unmapped field.
        """
        if not elastic_api.UNMAPPED_PREFIX:
            return field
 
        return f"{elastic_api.UNMAPPED_PREFIX}.{field}"
 

Here is the caller graph for this function:

◆ ingest()

GulpRequestStatus ingest	(		self,
		str	index,
		str	req_id,
		int	client_id,
		int	operation_id,
		str	context,
		str \| list[dict]	source,
		str	ws_id,
		GulpPluginParams	plugin_params = None,
		GulpIngestionFilter	flt = None,
		**	kwargs )

Ingests a file using the plugin.

NOTE: implementers should call super().ingest() in their implementation.
NOTE: this function *SHOULD NOT* raise exceptions

Args:
    index (str): name of the elasticsearch index to ingest the document into.
    req_id (str): The request ID related to this ingestion (must exist on the collab db).
    client_id (int): The client ID performing the ingestion.
    operation_id (int): The operation ID related to this ingestion.
    context (str): Context related to this ingestion.
    source (str|list[dict]): The path to the file to ingest, or a list of events dicts.
    ws_id (str): The websocket ID
    plugin_params (GulpPluginParams): additional parameters to pass to the ingestion function. Defaults to None.
    flt (GulpIngestionFilter, optional): filter to apply to this ingestion, if any. Defaults to None.
    kwargs: additional arguments if any

Definition at line 499 of file plugin.py.

    ) -> GulpRequestStatus:
        """
        Ingests a file using the plugin.
 
        NOTE: implementers should call super().ingest() in their implementation.
        NOTE: this function *SHOULD NOT* raise exceptions
 
        Args:
            index (str): name of the elasticsearch index to ingest the document into.
            req_id (str): The request ID related to this ingestion (must exist on the collab db).
            client_id (int): The client ID performing the ingestion.
            operation_id (int): The operation ID related to this ingestion.
            context (str): Context related to this ingestion.
            source (str|list[dict]): The path to the file to ingest, or a list of events dicts.
            ws_id (str): The websocket ID
            plugin_params (GulpPluginParams): additional parameters to pass to the ingestion function. Defaults to None.
            flt (GulpIngestionFilter, optional): filter to apply to this ingestion, if any. Defaults to None.
            kwargs: additional arguments if any
        """
        self.req_id = req_id
        self.client_id = client_id
        self.operation_id = operation_id
        self.context = context
        self.ws_id = ws_id
        self.index = index
        # raise NotImplementedError("not implemented!")
 

◆ ingest_plugin_initialize()

tuple[dict, GulpMapping] ingest_plugin_initialize	(		self,
		str	index,
		str \| dict	source,
		bool	skip_mapping = False,
		ProcessingPipeline	pipeline = None,
		str	mapping_file = None,
		str	mapping_id = None,
		GulpPluginParams	plugin_params = None )

Initializes the ingestion plugin.

Args:
    index (str): The name of the elasticsearch index.
    source (str|dict): The source of the record (source file name or path, usually. may also be a dictionary.).
    skip_mapping (bool, optional): Whether to skip mapping initialization (just prints source and plugin name, and return empty index mapping and custom mapping). Defaults to False.
    pipeline (ProcessingPipeline, optional): The psyigma pipeline to borrow the mapping from, if any. Defaults to None (use mapping file only).
    mapping_file (str, optional): name of the mapping file (i.e. 'windows.json') in the gulp/mapping_files directory. Defaults to None (use pipeline mapping only).
    mapping_id (str, optional): The mapping ID (options.mapping_id) in the mapping file, to get a specific mapping. Defaults to None (use first).
    plugin_params (GulpPluginParams, optional): The plugin parameters (i.e. to override mapping_file, mapping_id, ...). Defaults to None.
Returns:
    tuple[dict, GulpMapping]: A tuple containing the elasticsearch index type mappings and the enriched GulpMapping (or an empty GulpMapping is no valid pipeline and/or mapping_file are provided).

Definition at line 319 of file plugin.py.

    ) -> tuple[dict, GulpMapping]:
        """
        Initializes the ingestion plugin.
 
        Args:
            index (str): The name of the elasticsearch index.
            source (str|dict): The source of the record (source file name or path, usually. may also be a dictionary.).
            skip_mapping (bool, optional): Whether to skip mapping initialization (just prints source and plugin name, and return empty index mapping and custom mapping). Defaults to False.
            pipeline (ProcessingPipeline, optional): The psyigma pipeline to borrow the mapping from, if any. Defaults to None (use mapping file only).
            mapping_file (str, optional): name of the mapping file (i.e. 'windows.json') in the gulp/mapping_files directory. Defaults to None (use pipeline mapping only).
            mapping_id (str, optional): The mapping ID (options.mapping_id) in the mapping file, to get a specific mapping. Defaults to None (use first).
            plugin_params (GulpPluginParams, optional): The plugin parameters (i.e. to override mapping_file, mapping_id, ...). Defaults to None.
        Returns:
            tuple[dict, GulpMapping]: A tuple containing the elasticsearch index type mappings and the enriched GulpMapping (or an empty GulpMapping is no valid pipeline and/or mapping_file are provided).
 
        """
        # logger().debug("ingest_plugin_initialize: index=%s, pipeline=%s, mapping_file=%s, mapping_id=%s, plugin_params=%s" % (index, pipeline, mapping_file, mapping_id, plugin_params))
        logger().debug(
            "INITIALIZING INGESTION for source=%s, plugin=%s"
            % (muty.string.make_shorter(source, 260), self.name())
        )
        if skip_mapping:
            return {}, GulpMapping()
 
        # get path of the mapping file in gulp/mapping_files folder
        mapping_file_path = None
        if mapping_file is not None:
            mapping_file_path = gulp_utils.build_mapping_file_path(mapping_file)
 
        if plugin_params is not None:
            # override with plugin_params
            if plugin_params.mapping_file is not None:
                mapping_file_path = gulp_utils.build_mapping_file_path(
                    plugin_params.mapping_file
                )
            if plugin_params.mapping_id is not None:
                mapping_id = plugin_params.mapping_id
            if plugin_params.pipeline is not None:
                pipeline = plugin_params.pipeline
 
        index_type_mappings = await elastic_api.index_get_mapping(
            elastic_api.elastic(), index, False
        )
        # index_type_mappings = await elastic_api.datastream_get_mapping(self.elastic, index + '-template')
        m: GulpMapping = await mapping_helpers.get_enriched_mapping_for_ingestion(
            pipeline=pipeline,
            mapping_file_path=mapping_file_path,
            mapping_id=mapping_id,
        )
        return index_type_mappings, m
 

Here is the call graph for this function:

◆ internal()

bool internal ( self )

Returns whether the plugin is for internal use only

Definition at line 150 of file plugin.py.

    def internal(self) -> bool:
        """
        Returns whether the plugin is for internal use only
        """
        return False
 

◆ name()

str name ( self )

Returns the name of the plugin.

Definition at line 102 of file plugin.py.

    def name(self) -> str:
        """
        Returns the name of the plugin.
        """
 

Here is the caller graph for this function:

◆ options()

list[GulpPluginOption] options ( self )

return available GulpPluginOption list (plugin specific parameters)

Definition at line 77 of file plugin.py.

    def options(self) -> list[GulpPluginOption]:
        """
        return available GulpPluginOption list (plugin specific parameters)
        """
        return []
 

Here is the caller graph for this function:

◆ pipeline()

ProcessingPipeline pipeline	(		self,
		GulpPluginParams	plugin_params = None,
		**	kwargs )

Returns the pysigma processing pipeline for the plugin, if any.

Args:
    plugin_params (GulpPluginParams, optional): additional parameters to pass to the pipeline. Defaults to None.
    kwargs: additional arguments if any.
Returns:
    ProcessingPipeline: The processing pipeline.

Definition at line 538 of file plugin.py.

    ) -> ProcessingPipeline:
        """
        Returns the pysigma processing pipeline for the plugin, if any.
 
        Args:
            plugin_params (GulpPluginParams, optional): additional parameters to pass to the pipeline. Defaults to None.
            kwargs: additional arguments if any.
        Returns:
            ProcessingPipeline: The processing pipeline.
        """
        raise NotImplementedError("not implemented!")
 

Here is the caller graph for this function:

◆ query()

tuple[int, GulpRequestStatus] query	(		self,
		int	operation_id,
		int	client_id,
		int	user_id,
		str	username,
		str	ws_id,
		str	req_id,
		GulpPluginParams	plugin_params,
		GulpQueryFilter	flt,
		GulpQueryOptions	options = None )

used in query plugins to query data directly from external sources.

Args:
    operation_id (int): operation ID
    client_id (int): client ID
    user_id (int): user ID performing the query
    username (str): username performing the query
    ws_id (str): websocket ID to stream the returned data to
    req_id (str): request ID
    plugin_params (GulpPluginParams, optional): plugin parameters, including i.e. in GulpPluginParams.extra the login/pwd/token to connect to the external source, plugin dependent.
    flt (GulpQueryFilter): query filter (will be converted to the external source query format)
    options (GulpQueryOptions, optional): query options, i.e. to limit the number of returned records. Defaults to None.
        due to the nature of query plugins, not all options may be supported (i.e. limit, offset, ...) and notes creation is always disabled.
Returns:
    tuple[int, GulpRequestStatus]: the number of records returned and the status of the query.

Definition at line 119 of file plugin.py.

    ) -> tuple[int, GulpRequestStatus]:
        """
        used in query plugins to query data directly from external sources.
 
        Args:
            operation_id (int): operation ID
            client_id (int): client ID
            user_id (int): user ID performing the query
            username (str): username performing the query
            ws_id (str): websocket ID to stream the returned data to
            req_id (str): request ID
            plugin_params (GulpPluginParams, optional): plugin parameters, including i.e. in GulpPluginParams.extra the login/pwd/token to connect to the external source, plugin dependent.
            flt (GulpQueryFilter): query filter (will be converted to the external source query format)
            options (GulpQueryOptions, optional): query options, i.e. to limit the number of returned records. Defaults to None.
                due to the nature of query plugins, not all options may be supported (i.e. limit, offset, ...) and notes creation is always disabled.
        Returns:
            tuple[int, GulpRequestStatus]: the number of records returned and the status of the query.
        """
        return 0, GulpRequestStatus.FAILED
 

◆ record_to_gulp_document()

list[GulpDocument] record_to_gulp_document	(		self,
		int	operation_id,
		int	client_id,
		str	context,
		str	source,
		TmpIngestStats	fs,
		any	record,
		int	record_idx,
		GulpMapping	custom_mapping = None,
		dict	index_type_mapping = None,
		str	plugin = None,
		GulpPluginParams	plugin_params = None,
		**	kwargs )

Converts a record to one or more GulpDocument objects based on the provided index mappings.

Args:
    operation_id (int): The operation ID associated with the record.
    client_id (int): The client ID associated with the record.
    context (str): The context associated with the record.
    source (str): The source of the record (source file name or path, usually).
    fs (TmpIngestStats): The temporary ingestion statistics (may be updated on return).
    record (any): record to convert, plugin dependent format: note that here stacked plugins receives a list of GulpDocument objects instead (since the original record may generate one or more documents).
    record_idx (int): The index of the record in source.
    custom_mapping (GulpMapping, optional): The custom mapping to use for the conversion. Defaults to None.
    index_type_mapping (dict, optional): elastic search index type mappings { "ecs_field": "type", ... }. Defaults to None.
    plugin (str, optional): "agent.type" to be set in the GulpDocument. Defaults to None.
    plugin_params (GulpPluginParams, optional): The plugin parameters to use, if any. Defaults to None.
    extra (dict, optional): Additional fields to add to the GulpDocument (after applying mapping). Defaults to None.
    **kwargs: Additional keyword arguments:

Returns:
    list[GulDocument]: The converted GulpDocument objects or None if an exception occurred (fs is updated then).

Raises:
    NotImplementedError: This method is not implemented yet.

Definition at line 458 of file plugin.py.

    ) -> list[GulpDocument]:
        """
        Converts a record to one or more GulpDocument objects based on the provided index mappings.
 
        Args:
            operation_id (int): The operation ID associated with the record.
            client_id (int): The client ID associated with the record.
            context (str): The context associated with the record.
            source (str): The source of the record (source file name or path, usually).
            fs (TmpIngestStats): The temporary ingestion statistics (may be updated on return).
            record (any): record to convert, plugin dependent format: note that here stacked plugins receives a list of GulpDocument objects instead (since the original record may generate one or more documents).
            record_idx (int): The index of the record in source.
            custom_mapping (GulpMapping, optional): The custom mapping to use for the conversion. Defaults to None.
            index_type_mapping (dict, optional): elastic search index type mappings { "ecs_field": "type", ... }. Defaults to None.
            plugin (str, optional): "agent.type" to be set in the GulpDocument. Defaults to None.
            plugin_params (GulpPluginParams, optional): The plugin parameters to use, if any. Defaults to None.
            extra (dict, optional): Additional fields to add to the GulpDocument (after applying mapping). Defaults to None.
            **kwargs: Additional keyword arguments:
 
        Returns:
            list[GulDocument]: The converted GulpDocument objects or None if an exception occurred (fs is updated then).
 
        Raises:
            NotImplementedError: This method is not implemented yet.
        """
        raise NotImplementedError("not implemented!")
 

◆ sigma_plugin_initialize()

ProcessingPipeline sigma_plugin_initialize	(		self,
		ProcessingPipeline	pipeline = None,
		str	mapping_file = None,
		str	mapping_id = None,
		str	product = None,
		GulpPluginParams	plugin_params = None )

Initializes the Sigma plugin to convert sigma rules YAML to elasticsearch DSL query.

Args:
    pipeline (ProcessingPipeline, optional): The processing pipeline. Defaults to None (empty pipeline, plugin must fill it).
    mapping_file (str, optional): The name of the mapping file (i.e. 'windows.json') in the gulp/mapping_files directory. Defaults to None (use pipeline mapping only).
    mapping_id (str, optional): The mapping ID (["options"]["mapping_id"] in the mapping file, to get a specific mapping). Defaults to None (use first).
    product (str, optional): The product. Defaults to None.
    plugin_params (GulpPluginParams, optional): The plugin parameters, to override i.e. mapping_file_path, mapping_id, ... Defaults to None.
Returns:
    ProcessingPipeline: The initialized processing pipeline.

Definition at line 199 of file plugin.py.

    ) -> ProcessingPipeline:
        """
        Initializes the Sigma plugin to convert sigma rules YAML to elasticsearch DSL query.
 
        Args:
            pipeline (ProcessingPipeline, optional): The processing pipeline. Defaults to None (empty pipeline, plugin must fill it).
            mapping_file (str, optional): The name of the mapping file (i.e. 'windows.json') in the gulp/mapping_files directory. Defaults to None (use pipeline mapping only).
            mapping_id (str, optional): The mapping ID (["options"]["mapping_id"] in the mapping file, to get a specific mapping). Defaults to None (use first).
            product (str, optional): The product. Defaults to None.
            plugin_params (GulpPluginParams, optional): The plugin parameters, to override i.e. mapping_file_path, mapping_id, ... Defaults to None.
        Returns:
            ProcessingPipeline: The initialized processing pipeline.
        """
        logger().debug("INITIALIZING SIGMA plugin=%s" % (self.name()))
 
        mapping_file_path = None
        if mapping_file is not None:
            # get path in mapping directory
            mapping_file_path = gulp_utils.build_mapping_file_path(mapping_file)
 
        if plugin_params is not None:
            # override with plugin_params
            if plugin_params.mapping_file is not None:
                mapping_file_path = gulp_utils.build_mapping_file_path(
                    plugin_params.mapping_file
                )
            if plugin_params.mapping_id is not None:
                mapping_id = plugin_params.mapping_id
 
        p = await mapping_helpers.get_enriched_pipeline(
            pipeline=pipeline,
            mapping_file_path=mapping_file_path,
            mapping_id=mapping_id,
            product=product,
        )
        return p
 

Here is the call graph for this function:

◆ tags()

list[str] tags ( self )

returns a list of tags for the plugin. Tags are used to aid filtering of plugins/query filters in the UI.
- "event"
- "network"
- "file"
- "process"
- "threat"
- "threat.enrichments"
- ...

Definition at line 156 of file plugin.py.

    def tags(self) -> list[str]:
        """
        returns a list of tags for the plugin. Tags are used to aid filtering of plugins/query filters in the UI.
        - "event"
        - "network"
        - "file"
        - "process"
        - "threat"
        - "threat.enrichments"
        - ...
        """
        return []
 

◆ type()

GulpPluginType type ( self )

Returns the plugin type.

Definition at line 84 of file plugin.py.

    def type(self) -> GulpPluginType:
        """
        Returns the plugin type.
        """
 

Here is the caller graph for this function:

◆ version()

str version ( self )

Returns plugin version.

Definition at line 90 of file plugin.py.

    def version(self) -> str:
        """
        Returns plugin version.
        """
 

Member Data Documentation

◆ buffer

list buffer = []

Definition at line 71 of file plugin.py.

◆ client_id

str client_id = None

Definition at line 65 of file plugin.py.

◆ context

str context = None

Definition at line 67 of file plugin.py.

◆ index

str index = None

Definition at line 64 of file plugin.py.

◆ operation_id

str operation_id = None

Definition at line 66 of file plugin.py.

◆ path

path = path

Definition at line 69 of file plugin.py.

◆ req_id

str req_id = None

Definition at line 63 of file plugin.py.

◆ ws_id

str ws_id = None

Definition at line 68 of file plugin.py.

The documentation for this class was generated from the following file:

src/gulp/plugin.py

Public Member Functions

Public Attributes

Protected Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ _build_gulpdocuments()

◆ _build_ingestion_chunk_for_ws()

◆ _call_record_to_gulp_document_funcs()

◆ _finish_ingestion()

◆ _flush_buffer()

◆ _ingest_record()

◆ _map_source_key()

◆ _parser_failed()

◆ _process_plugin_params()

◆ _process_record()

◆ _record_failed()

◆ _type_checks()

◆ cleanup()

◆ depends_on()

◆ desc()

◆ event_type_field()

◆ get_unmapped_field_name()

◆ ingest()

◆ ingest_plugin_initialize()

◆ internal()

◆ name()

◆ options()

◆ pipeline()

◆ query()

◆ record_to_gulp_document()

◆ sigma_plugin_initialize()

◆ tags()

◆ type()

◆ version()

Member Data Documentation

◆ buffer

◆ client_id

◆ context

◆ index

◆ operation_id

◆ path

◆ req_id

◆ ws_id

◆ init()