[Midnightbsd-cvs] [MidnightBSD/mports] 6d8bd8: Refactor Magus pipeline phases (#604)

[Lucas Holt]

  Branch: refs/heads/master
  Home:   https://github.com/MidnightBSD/mports
  Commit: 6d8bd8c4d39d43e25c55109416189187b646adb8
      https://github.com/MidnightBSD/mports/commit/6d8bd8c4d39d43e25c55109416189187b646adb8
  Author: Lucas Holt <luke at foolishgames.com>
  Date:   2026-06-02 (Tue, 02 Jun 2026)

  Changed paths:
    M Tools/lib/Magus.pm
    M Tools/lib/Magus/App/Slave/Worker.pm
    M Tools/lib/Magus/Config.pm
    M Tools/lib/Magus/Index.pm
    M Tools/lib/Magus/Lock.pm
    A Tools/lib/Magus/Phase.pm
    A Tools/lib/Magus/PhaseLog.pm
    A Tools/lib/Magus/PhaseResult.pm
    M Tools/lib/Magus/Port.pm
    M Tools/lib/Magus/PortTest.pm
    M Tools/lib/Magus/Run.pm
    M Tools/magus/config.yaml.example
    M Tools/magus/distcheck/distcheck.cxx
    M Tools/magus/master/delete_run.pl
    A Tools/magus/phased-pipeline-plan.md
    M Tools/magus/ready_ports.sql
    M Tools/magus/schema.sql
    M Tools/magus/slave/magus.pl
    M Tools/magus/www/data/magus/auth/mcp.cgi
    M Tools/magus/www/data/magus/index.cgi
    M Tools/magus/www/tmpls/index.tmpl
    M Tools/magus/www/tmpls/port-list.tmpl
    M Tools/magus/www/tmpls/port.tmpl
    M Tools/magus/www/tmpls/run.tmpl
    M ports-mgmt/magus-utils/Makefile

  Log Message:
  -----------
  Refactor Magus pipeline phases (#604)

## Summary
- split Magus work into independent fetch, build, test, and virus scan
phases
- keep ports.status as the build/package status so magus bless ignores
test and scan results
- add phase-aware locks, phase result/log tables, slave scheduling,
web/API phase display, and ClamAV scanner configuration
- add ClamAV to ports-mgmt/magus-utils for slave node installs

## Validation
- git diff --check
- targeted Perl syntax checks for modified Magus modules and CGI/slave
scripts with DB/config stubs where needed
- bmake in Tools/magus/distcheck
- distfile path normalization probe for
grouped/query/subdirectory/unsafe paths

## Notes
- schema/runtime database load was not verified locally because no
PostgreSQL server was listening on /tmp/.s.PGSQL.5432 during development
- unrelated untracked files are present in the local worktree and were
not included in this branch

## Summary by Sourcery

Refactor Magus to support phase-aware fetch, build, test, and virus scan
pipelines while preserving build status semantics, with corresponding
scheduler, database, web/API, and slave worker updates.

New Features:
- Introduce per-phase results and logs for fetch, build, test, and scan,
including ClamAV-based virus scanning of built packages.
- Expose per-phase status and logs via the web UI, APIs, and CLI tooling
while keeping overall port status as the build result.
- Allow workers to be configured to process specific phases and schedule
ports per phase independently of the main build pipeline.

Bug Fixes:
- Normalize and validate distfile paths to safely handle grouped,
query-string, and subdirectory filenames, creating necessary directories
before copying.

Enhancements:
- Make locks phase-aware and adjust ready-port selection, dependency
handling, and run-emptiness checks to operate on build vs non-build
phases correctly.
- Extend schema, indices, and foreign keys to support per-phase results,
logs, timestamps, and machine attribution without changing existing logs
semantics.
- Refine event recording to include phase and structured names, and
update log handling so build logs remain backward compatible while
additional phase logs are tracked separately.
- Update distfile handling and helper tools to preserve directory
structure when copying into cache locations.

Build:
- Add ClamAV as a runtime dependency to magus-utils for virus scanning
support.

Documentation:
- Document the phased pipeline design and behavior in a new Magus phased
pipeline refactor plan.

Tests:
- Enhance the distcheck helper to validate that distfiles with
subdirectory paths are copied correctly.

---------

Signed-off-by: Lucas Holt <luke at foolishgames.com>



To unsubscribe from these emails, change your notification settings at https://github.com/MidnightBSD/mports/settings/notifications