3.12.3 because of the issue
#612.normalizer processor, as it’s functionality was replaced by the grokker, timestamper and field_manager processorstimestamper on ISO8601 and UNIX time parsingfield_manager where extending a non list target leads to a processing failurepseudonymizer where a missing regex_mapping from an existing config_file causes logprep to crash continuouslypseudonymizer change rule config field pseudonyms to mappingclusterer change rule config field target to source_fieldsgeneric_resolver change rule config field append_to_list to extend_target_listhyperscan_resolver change rule config field append_to_list to extend_target_listcalculator now adds the error tag _calculator_missing_field_warning to the events tag field instead of _calculator_failure in case of missing field in eventsdomain_label_extractor now writes _domain_label_extractor_missing_field_warning tag to event tags in case of missing fieldsgeoip_enricher now writes _geoip_enricher_missing_field_warning tag to event tags in case of missing fieldsgrokker now writes _grokker_missing_field_warning tag to event tags instead of _grokker_failure in case of missing fieldsrequester now writes _requester_missing_field_warning tag to event tags instead of _requester_failure in case of missing fieldstimestamp_differ now writes _timestamp_differ_missing_field_warning tag to event tags instead of _timestamp_differ_failure in case of missing fieldstimestamper now writes _timestamper_missing_field_warning tag to event tags instead of _timestamper_failure in case of missing fields--thread_count parameter to --thread-count in http generator--report parameter and feature from http generatorextend_target_list in the field managerthe ordering of the given source fields is now preservedrestart_count parameter is configurable. If you want the old behavior back, you can set this parameter to a negative numberlogprep pseudo --helprestart_count parameter to configurationmode to pseudonymizer processor and to pseudonymization tools to chose the AES Mode for encryption and decryptionrejected_execution_exceptionwrite_to_target, add_field_to and get_dotted_field_value when reading and writing from and to events
hyperscan_resolver, generic_resolver and template_replacerpseudonymizer, pre_detector, selective_extractor processors and pipeline to handle extra_data the same wayclusterer, pre_detector and pseudonymizer processors and change rule_tree so that the processor do not require process override
clusterer_handle_missing_fields from the field_managerLogprepMPQueueListener to outsource logging to a separate processQueuehandler to root logger to ensure all logs were handled by LogprepMPQueueListenerhttp_generator to use a logprep http output connectorcached_properties are populated during setup time--username and --password parameters optional in http generatorFileNotFoundError is raised during processinghttp_input connectorhostname available in custom log formatslogprep.event_generator module to logprep.generatoropensearch and elasticsearchhttp_input connector
nummer_of_http_requestsmessage_backlog_sizeuser_manual/security/best_practices.html
versioneer dependency in favor of setuptools-scmgeneric_adder not creating new field from type listAutoRuleCorpusTester for logprep test integrationtest unit), which was broken after adding support for multiple configuration files and resolving paths in configuration filessetup.py that corrupted the installlogprep --help for more information.SIGUSR1logprep test confignumber_of_successful_writes metric to the s3 connector, which counts how many events were successfully written to s3_write_backlog method introduced by the confluent_kafka commit bugfix in v9.0.0logprep print json|yaml <Path to config> in json or yamlFatalOutputError instead of warningsbatch_finished_callback to prevent data loss that could be caused by partially processed eventsschema_and_rule_checker modulefastapi and aiohttp to address CVEsmax_retries parameterFatalOutputError on handling CriticalInputError in pipelinethread_count, queue_size and chunk_size configurable for parallel_bulk in opensearch output connectorparallel_bulk implementation not delivering messages to opensearchLOGPREP_CONFIG_AUTH_USERNAME=<your_username>LOGPREP_CONFIG_AUTH_PASSWORD=<your_password>LOGPREP_CONFIG_AUTH_TOKEN=<your_token>LOGPREP_CONFIG_AUTH_METHOD=oauthpseudonymizer processor
lru_cachemax_caching_days config optionmax_cached_pseudonymized_urls config option which defaults to 1000confluent_kafka commit issue after kafka did some rebalancing, fixes also negative offsetsid could possibly break configurations if the same rule is used in both rule trees
id to each rule or delete the possibly redundant rulecalculator processor with new added function from_hexquickstart/exampledata/config/grafana/dashboardsid for all rules to identify rules in metrics and logs
id is given, the id will be generated in a stable wayid uniqueness on processor level over both rule trees to ensure metrics are counted correctly on rule levelDEBUGkafka_config dictionary for confluent_kafka_input and confluent_kafka_output connectorsgeneric_rules or specific_rulesfield_manager to ignore missing source fields to suppress warnings and failure tagscalculator, concatenator, dissector, grokker, ip_informer, selective_extractorenable.auto.commit: falsepre_detector processor now adds the field creation_timestamp to pre-detections.
It contains the time at which a pre-detection was created by the processor.prometheus and grafana to the quickstart setup to support developmentCookie HTTP header isn’t stripped on cross-origin redirectsFix writing time measurements into the event after the deleter has deleted the event. The bug only
happened when the metrics.measure_time.append_to_event configuration was set to true.
Fix memory leak by removing the log aggregation capability
DummyInput, JsonInput,
JsonlInput. This enables easier debugging by introducing a continues input stream of documents.system_%{type}Grokker by introducing a timeout limit of one secondrule_filter will stay
in place of the lucene_filter name.amides.confidence field.
In case of positive detection results, rule attributions are now inserted in the amides.attributions field.UNIXPATH by internally converting [[:alnum:]] to \w"dry_runner to support extra outputs of the selective_extractorPROMETHEUS_MULTIPROC_DIR environment variable optional, will default to
/tmp/PROMETHEUS_MULTIPROC_DIR if not givenrequests to >=2.31.0 to circumvent CVE-2023-32681%{field-( )}
notation.FieldManager such that it can move/copy multiple source fields into multiple targets
inside one rule.timestamper processor to extract timestamp functionality from normalizerarrow dependency and depending features for performance reasons
datetime.strftime syntax in timestamp_differ, s3_output, elasticsearch_output and opensearch_outputlogprep.util.time.TimeParseramides processor to extends conventional rule matching by applying machine learning componentsgrokker processor to extract grok functionality from normalizerNormalizer writes failure tags if nomalization failsflush_timeout to opensearch and elasticsearch outputs to ensure message delivery within a configurable periodkafka_config option to confluent_kafka_input and confluent_kafka_output connectors to provide additional config options to librdkafkajson to msgspec in pipeline and kafka connectorsauto_rule_corpus_tester by initializing all processors
between test cases.None valuesv4.0.0selective_extractor, pseudonymizer, pre_detector to support multiple outputsstring_splitter processor to split strings of variable length into listsip_informer processor to enrich events with ip informationPipeline in python without input/output connectorsauto_rule_corpus_tester to test a whole rule corpus against defined expected outputs.dissector dissect pattern languageattrs to >=22.2.0 and delete redundant min_len_validatorlogprep_pipeline_number_of_warnings to
logprep_pipeline_sum_of_processor_warnings and logprep_pipeline_number_of_errors to
logprep_pipeline_sum_of_processor_errors)opensearch_output and elasticsearch_output not handling authentication issueslogprep_pipeline_number_of_processed_events to actually count the processed events per pipelineLOGPREP_, CI_, GITHUB_ or PYTEST_selective_extractor3.6, 3.7, 3.8dissectorhttp input connector that spawns a uvicorn server which parses requests content to events.file input connector that reads generic logfiles.requester processor that enriches by making http requests with field valuescalculator processor to calculate with or without field valuesgeoip_enricher configurable by introducing the rule config
customize_target_subfieldstimestamp_differ processor that can parse two timestamps and calculate their respective time delta.config_refresh_interval configuration option to refresh the configuration on a given timedeltadissector to use a prefix pattern in dissect language for appending to strings and add the default behavior to append to strings without any prefixed separator3.10 and 3.11list_search_base_path config parameter in list_comparison processorgeoip_enricher to download the geoip-databasenumber_of_processed_events metric in input connector. Will now only count actual
events.connector config into input and output to compose connector config independendlypreprocessing of the input processordelete processor was renamed to deleterwriting_output connector to jsonl_outputinput and outputconcatenator processor that can combine multiple source fieldsdissector processor that tokinizes messages into new or existing fieldskey_checker processor that checks if all dotted fields from a list are present in the eventfield_manager processor that copies or moves fields and merges listsconcatenator, datetime_extractor, dissector, domain_label_extractor, domain_resolver, geoip_enricher and list_comparisondatetime_extractor, domain_label_extractor, domain_resolver, geoip_enricher and list_comparisonsphinxcontrib.datatemplates and testcase-renderer to docsget_dotted_field_value helper method which should lead to increased performancedropper processor code to improve performancedatetime_extractor.datetime_field is deprecated. Use datetime_extractor.source_fields as list instead.datetime_extractor.destination_field is deprecated. Use datetime_extractor.target_field instead.delete is deprecated. Use deleter.delete instead.domain_label_extractor.target_field is deprecated. Use domain_label_extractor.source_fields as list instead.domain_label_extractor.output_field is deprecated. Use domain_label_extractor.target_field instead.domain_resolver.source_url_or_domain is deprecated. Use domain_resolver.source_fields as list instead.domain_resolver.output_field is deprecated. Use domain_resolver.target_field instead.drop is deprecated. Use dropper.drop instead.drop_full is deprecated. Use dropper.drop_full instead.geoip_enricher.source_ip is deprecated. Use geoip_enricher.source_fields as list instead.geoip_enricher.output_field is deprecated. Use geoip_enricher.target_field instead.label is deprecated. Use labeler.label instead.list_comparison.check_field is deprecated. Use list_comparison.source_fields as list instead.list_comparison.output_field is deprecated. Use list_comparison.target_field instead.pseudonymize is deprecated. Use pseudonymizer.pseudonyms instead.url_fields is deprecated. Use pseudonymizer.url_fields instead.number_of_matches.failure_target_field is specified in the configurationversion_info_target_field.connector > consumer > preprocessing configurationjsonpreprocessing, maintain backward compatibility,
but mark old version as deprecated.--version cli argument to print the current logprep version, as well as the configuration
version if foundprocessor_stats to metrics. Metrics are now collected in separate dataclassesmetrics instead of status_loggerlogprep.abc.processor.Processordelete processor to be used with rules.donothing processorattrs based Config classes for each processor__slots__ProcessorRegistry to register all processorsProcessorConfiguration as an adapter to create configuration for processorslogprep.processor.processor_factory.ProcessorFactoryProcessorFactorytld_lists for domain_resolver, domain_label_extractor, pseudonymizerdomain_resolver to make code cleaner and increase test coverageujson dependency because of CVE