Makefile: Automatically add all tools/ to POTFILE.in

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Fedora no longer seems to install awk by default

This is needed to run the configure script.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Ensure loc_database_lookup finds a match

Commit 9e72b8a modified the lookup function to return 0 even if no
matches are found. As result, the test case could pass even if the
database lookup did not find any matches.

This commit restores the original intent of the test case, by adding
a failure condition when `network1` is NULL. This is similar to how
the subsequent test case was modified for the inverse condition.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "jenkins: Build Debian packages for riscv64, too"

This reverts commit 92bfc43af92eb19a5ed95ff04f2a71e56087a75b.

Well, this did not last very long...

+ apt-get install -y apt-utils build-essential crossbuild-essential-riscv64 devscripts qemu-user-static
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package crossbuild-essential-riscv64
script returned exit code 100

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Build Debian packages for riscv64, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Fix passing custom LDFLAGS

Fixes: #13829 - Overlinking Lua and Python bindings
Reported-by: Petr Pisar <ppisar@redhat.com>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Fix indentation

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Update libloc1.symbols for version 0.9.18

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Okay, drop Alma Linux again

This thing just seems to be too ancient and does not have packages for
Lua. Instead of creating too many if/else blocks to cover this, let's
remove it again. As far as I am aware, we don't have any known users on
this distribution.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Alma Linux does not seem to understand Code Ready

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Fix installing development tools on Alma Linux

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Try building for Alma Linux 9

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Build for s390x again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Fix endianess problem when fetching octets in IPv4 addresses

Fixes: #13828
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Publish Debian packages for all supported architectures

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Bump version to 0.9.18

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Remove broken architectures for now

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Correctly check return value of dup()

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Remove dead assignment

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix string formatting issues on 32 bit systems

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Package Lua files

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Update filelists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Build against Lua again

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Don't build man pages

This is not great, but I currently don't have any better option.
Installing asciidoc conflicts with python3-all-dev when cross-compiling.
Since asciidoc is optional and Python isn't this at least generates a
package at the end.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix unintended fallthrough

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

python: Fix type for keyword lists

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Fix syntax to install perl(Test::More)

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Perform formatting string sanitization

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Replace bitfields with flags to mark deleted nodes

clang does not seem to like these much.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

log: Perform formatting string sanitation when logging to stderr

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Install perl(Test::More) on Fedora

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Constify path to open in database test

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Never pass zero to __builtin_ctz

This is undefined behaviour which worked just fine in GCC, but in clang
the code is doing funny business.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Enable more compiler warnings

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Fix all sorts of string formatting issues

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Make bit length mismatch message clearer

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Upload logs from tests in src/, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Try harder to merge networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tree: Add network to the stack after we have tried again

If we add it before, we will try to merge the network with itself which
is not necessary as it will never result in anything good.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Make Lua check work on Fedora, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Install lua-unit wherever available

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Don't run Lua tests if luaunit is not available

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

debian: Don't build packages against Lua

This is a little bit more complicated, because we probably want to build
against multiple versions and I don't know how to do this.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Actually install Lua when we want to build against it

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Add all supported Debian architectures

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Fail if Lua was enabled, but not found

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Remove extra tests we currently don't support

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Don't expect any tests to fail

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Always Lua extension, too

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Check syntax of Lua check

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Don't automatically detect systemdunitdir

This is no longer possible on newer versions of systemd where the
variable has been removed from pkg-config.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

jenkins: Initial import

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

data: Import today's database

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

po: Update POTFILES.in

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Return nothing in case invalid data has been passed to libloc

The old croach approch instantly will stop the script if invalid data
will be passed to libloc and also will fail to proper execute the
testsuite.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Don't import /4 or /10 networks from the routing table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Ignore any sudden disconnects when we fetch a Geofeed

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Update a few AWS locations

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

perl: Fix a couple of NULL-pointer derefences in the module

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

libloc: Refactor summarizing IP address ranges

The algorithm seems to have had some bugs where it generated too many
unnecessary subnets which caused the system to run out of memory very
quickly when determining bogons.

This version of the algorithm is now a little bit smarter and should be
significantly faster as well.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: lua: Set a variable to true if we are checking for a boolean later

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Ensure that the testsuite is being executed with the correct version

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Check if we got returned something on some tests

Since Lua does not know about exceptions, functions usually return nil
instead of some value.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Ensure that we set timestamps in the announcements table

Otherwise aging out older announcements does not work.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Fix raising an exception if no network was found

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Have the lookup function return 0 even if nothing was found

This is a potentially breaking change, but I think we need to do this as
the previous return code was not very descriptive.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Implement setting a log callback function

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

libloc: Allow passing a pointer to the log callback

This is basically re-implemnting the logic to pass log settings as the
previous functions were inappropriately named.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Revert "database: Re-open the file handle in r+ mode"

This reverts commit 7654a5beb9b46758af613f15e01f8e0d0b3bd730.

This change breaks the library on Linux.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Initialize the database object pointer

Unfortunately Lua does not initialize any new user data. Therefore it
could happen that the cleanup function will try to tidy up after a new
Database object could not be set up correctly and cause a segmentation
fault.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Scan for multiple Lua versions

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

database: Re-open the file handle in r+ mode

On Mac OS, we cannot re-read our temporary file from the testsuite unless we re-open in r+ mode.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Move the cursor back to end when finished writing

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

writer: Fail if the header could not be written successfully

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

Makefile: Don't try to use -export-symbols if --version-script= isn't available

libtool seems to mess up our symbol table by prepending an underscore character
to all lines which renders the file unparsable.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

configure: Use pkg-config to find OpenSSL

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop EDROP as it has been merged into DROP

http://www.spamhaus.org/resource-hub/network-security/spamhaus-drop-and-edrop-to-become-a-single-list/

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add method to access database creation time

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add function that returns subnets of a network

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Add function to return a reverse pointer for networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Set LD_LIBRARY_PATH

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Add functions to access a specific byte/nibble in an address

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Cleanup any database iterators

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Create a simple iterator for all networks

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Fix bit length tests

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Don't raise an error if a network cannot be found

The lookup function now returns nil which is easier to handle than
catching a Lua error.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

lua: Add compatibility function to compile with Lua >= 5.1

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

tests: Add tests for #13236

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

network: Fix handling bit length on merge

The check was kind of useless because of incorrect values from the bit
length function.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

address: Fix bit length calculation

I have no idea what the previous version computed, but it wasn't the
right thing.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Permit Geofeeds for everything instead of ignoring

This seems to become the default and so we should avoid making the
overrides file too verbose.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop even more indexes

We have gone a little bit wild on this but should not need them.

This patch also migrates any GIST indexes to SP-GIST.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Drop any indexes we no longer need

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Make the export 200x faster

The export became a huge problem here as it was very slow. One large
query produced one very large output which was difficult to test and
validate as the total export could easily take 17-20 hrs.

This patch fixes this so that the export now takes around ~5min. Sadly
we are still not using more than one processor core, because PostgreSQL
thinks its faster with just the one, but we trade a lot of CPU time for
a little bit of extra memory instead.

We create a new temporary table that holds all networks that we are
interested in. We then run a couple of broken down queries that perform
one thing instead of multiple at the same time.

First, we add all AS numbers, then all countries which are the longest
queries taking now around 20-30s. Then we apply any data from the
external feeds and our own overrides.

The final export requires about 1.7 GiB of memory because we don't
perform any deduplication in the database any more. That has been a lot
better implemented in our own code and I don't think that < 2 GiB is a
reason for concern.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Replace all GIST indexes with SP-GIST

They are a lot faster for our application.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: ANALYZE all tables before we are running the export

This should help the query planner to make better decisions when
planning the large queries.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Remove more traces of the Geofeed overrides table

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Convert networks back to string

psycopg3 returns inet types as IP networks.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Allow storing multiple Geofeeds for the same network

This is very likely not required except for a couple of overrides.

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>

importer: Currently update the source when encountering a conflict

Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>