FreeDV
/
freedv-gui
mirror of https://github.com/drowe67/freedv-gui.git
1
0
Fork 0
Code Issues Releases Wiki Activity

Windows CI: RADE loss testing + ASan/UBSan sanitizer coverage (#1487) 

* CI (Windows): add RADE loss testing (prerequisite for sanitizer coverage)

Ports the RADE loss test infrastructure from the ms-rade-v2 branch
(as of commit f013450b there) so the Windows sanitizer jobs added in
the following commits have a RADE loss test to actually run:

- test/TestFreeDVRadeLoss.ps1: transmits a known corpus through FreeDV,
  records the result, plays it back through RX, and compares TX/RX RADE
  features via loss.py against a threshold.
- test/RadeVerificationReport.ps1: assembles the RADE integration
  verification report from a test run.
- cmake-windows.yml: adds the rade-loss-baseline job (computes the loss
  threshold from a software-only baseline on Linux, since rade_tx_wav/
  rade_rx_wav are excluded from the Windows build), wires RADE loss
  testing and verification-report generation into the `test` job, adds
  crash dump collection (WER LocalDumps) for post-mortem diagnosis, and
  switches audio-endpoint readiness waiting to Wait-AudioDevices.ps1 to
  avoid starting a test before a virtual cable is actually enumerable.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jz1Nz4hmnEzwQ7hjQtCBhk

* Windows CI: add UBSan/ASan sanitizer build+test jobs

llvm-mingw supports UBSan on every architecture it targets and ASan on
x86 only (confirmed against the toolchain's own release assets: only
x86_64/i386 ship a libclang_rt.asan_dynamic-*.dll, no aarch64 variant
exists). Adds build-sanitizer (mirrors build-pgo-inst minus PGO, one
variant per {UBSAN x86_64, UBSAN aarch64, ASAN x86_64}) and
test-sanitizer (mirrors test, pointed at the new artifacts).

No manual DLL copy step needed for ASan: cmake/GetDependencies.cmake.in
already walks freedv.exe's PE import table via objdump and derives the
toolchain's <arch>-w64-mingw32/bin/ directory as a search path when
FREEDV_USING_LLVM_MINGW is set -- the same mechanism that bundles every
other DLL dependency already picks up libclang_rt.asan_dynamic-*.dll
automatically once ENABLE_ASAN is on. This path was already prepared
for in a prior commit (73cd6f49, "Fix cross-compile definitions to
allow asan to be used in the first place") but never wired into CI.

Sanitizer test steps are continue-on-error, matching the existing
SANITIZERS_ENABLED leniency on macOS/Linux (ctest doesn't enforce the
PASS_REGULAR_EXPRESSION under sanitizers there either) -- a sanitizer
build is slower and less reliable for real-time audio, so what matters
here is whether a sanitizer catches a genuine memory-safety/UB bug
(an abort with a diagnostic in the log), not the loss threshold.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jz1Nz4hmnEzwQ7hjQtCBhk

* Fix Windows sanitizer options: drive-letter colon breaks the parser

Every test-sanitizer run was failing at freedv.exe startup with
"AddressSanitizer: ERROR: expected '=' in ASAN_OPTIONS" (and the UBSan
equivalent), masked as job "success" by continue-on-error -- meaning
zero actual testing happened in the previous push.

ASAN_OPTIONS/UBSAN_OPTIONS are colon-separated key=value pairs. The
absolute path used for suppressions= started with a Windows drive
letter (D:\...), and the sanitizer's own option parser split on that
colon too: "suppressions=D" parsed as one pair, then "\...\foo.txt"
(no '=') aborted the parse before FreeDV ever started.

Fix: copy the suppression files next to freedv.exe (alongside the
existing test script copies) and reference them by bare relative
filename, which contains no colon to collide with.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jz1Nz4hmnEzwQ7hjQtCBhk

* Suppress known wxWidgets MSW UBSan finding (null pointer in tooltip.cpp)

wxToolInfo's constructor does a member access through a null pointer of
type TTTOOLINFOW in wx's own MSW backend (tooltip.cpp:100), hit
consistently on both new Windows UBSAN test-sanitizer jobs
(windows-2022 and windows-11-arm). Third-party code, not ours -- same
treatment as the existing macOS vptr suppressions for wx's own latent
UB.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jz1Nz4hmnEzwQ7hjQtCBhk

* Suppress libressl UBSan finding; make Windows sanitizer jobs actually fail

test/ubsan_suppressions.txt: add function:crypto/stack/stack.c for
"call to function local_sk_X509_NAME_ENTRY_pop_free through pointer to
incorrect function type" -- a well-known, benign pattern in OpenSSL/
LibreSSL's type-erased sk_TYPE_pop_free callback casting, not a real bug.

cmake-windows.yml: the three test-sanitizer test steps are
continue-on-error (needed so an ordinary loss-threshold miss under
instrumentation overhead doesn't stop the other tests, matching the
SANITIZERS_ENABLED leniency ctest already gives sanitizer builds on
macOS/Linux) -- but continue-on-error suppresses the job's conclusion,
not just the step's outcome, so a genuine sanitizer abort was *also*
being swallowed into job "success" with no fix. Added a "Check for
sanitizer errors" step (no continue-on-error) that scans every test's
captured output for an actual AddressSanitizer/UndefinedBehaviorSanitizer/
LeakSanitizer report and fails the job for real if one is found, while
a plain "Test failed" from a loss-threshold miss (no sanitizer report
in the output) still doesn't. Also wired Tee-Object logging into the
FullDuplex and Reporting test steps (RadeLoss already had it) so all
three are covered by the scan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jz1Nz4hmnEzwQ7hjQtCBhk

* CI (Windows): drop RadeVerificationReport, fix missing hamlibserver.py

Remove RadeVerificationReport.ps1 and the workflow steps that generate/
upload the RADE verification report from the `test` job -- out of scope
for this PR (Windows ASan/UBSan CI + the RADE loss test infra it needs).

Separately: PR CI showed both `test` and `test-sanitizer (ASAN, ...)`
failing the RADE Reporting step with "python.exe: can't open file
'...\bin\hamlibserver.py'". TestFreeDVReporting.ps1 unconditionally
launches hamlibserver.py (a mock rigctld) as a subprocess before
starting FreeDV, but neither job's "Copy test scripts to install
folder" step actually copied it there -- only TestFreeDVReporting.ps1
itself and its conf template were copied. Added the missing Copy-Item
to both jobs.

(The consequent "Couldn't connect to Radio with hamlib" fatal error
also triggered an apparent AddressSanitizer heap-use-after-free in
wxMutexInternal::LockTimeout during the ASan job's abnormal shutdown --
very likely collateral damage from the broken test flow rather than a
real bug, but worth re-checking once this fix lands.)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jz1Nz4hmnEzwQ7hjQtCBhk

* CI (Windows): actually commit the RadeVerificationReport/hamlib workflow changes

Follow-up to 54aa6988 -- that commit only picked up the file deletion;
the cmake-windows.yml edits (removing the report generation/upload
steps and the RadeVerificationReport.ps1 copy, adding the missing
hamlibserver.py copy to both the test and test-sanitizer jobs) didn't
get staged due to a failed `git add` on the already-removed path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jz1Nz4hmnEzwQ7hjQtCBhk

* master still uses RADEV1, not V2.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
v3.0-dev
Mooneer Salem 2026-09-13 10:53:43 -07:00 committed by GitHub
parent 07a51dc6ca
commit 2c7aa5ff82
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 841 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

483
.github/workflows/cmake-windows.yml vendored
View File

@ -81,6 +81,147 @@ jobs:
path: ${{github.workspace}}/build_windows/instrumented-${{ steps.installer-filename.outputs.INSTALLER_FILENAME }}
archive: false
# Sanitizer coverage for the Windows builds. llvm-mingw supports UBSan on
# every architecture it targets and ASan on x86 only (no aarch64 ASan
# runtime is shipped) -- see https://github.com/mstorsjo/llvm-mingw#readme.
# Not PGO-optimized: these exist purely to catch memory-safety/UB bugs
# that a plain Release build's counters can't see, not to represent
# production performance.
build-sanitizer:
strategy:
fail-fast: false
matrix:
include:
- sanitizer_name: UBSAN
sanitizer_flag: -DENABLE_UBSAN=1
arch: x86_64
- sanitizer_name: UBSAN
sanitizer_flag: -DENABLE_UBSAN=1
arch: aarch64
- sanitizer_name: ASAN
sanitizer_flag: -DENABLE_ASAN=1
arch: x86_64
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v6
- name: ccache
uses: hendrikmuhs/ccache-action@v1.2.23
with:
key: ${{ github.job }}-${{ matrix.sanitizer_name }}-${{ matrix.arch }}
max-size: "5G"
verbose: 2
- name: Install required packages
shell: bash
working-directory: ${{github.workspace}}
run: |
sudo apt-get update
sudo apt-get install nsis autoconf automake libtool libltdl-dev pkg-config
- name: Download MinGW LLVM
shell: bash
working-directory: ${{github.workspace}}
run: |
wget https://github.com/mstorsjo/llvm-mingw/releases/download/20251216/llvm-mingw-20251216-ucrt-ubuntu-22.04-x86_64.tar.xz
tar xvf llvm-mingw-20251216-ucrt-ubuntu-22.04-x86_64.tar.xz
- name: Configure freedv-gui (unsigned, ${{ matrix.sanitizer_name }})
shell: bash
working-directory: ${{github.workspace}}
run: |
export PATH=${{github.workspace}}/llvm-mingw-20251216-ucrt-ubuntu-22.04-x86_64/bin:$PATH
mkdir build_windows
cd build_windows
cmake ${{ matrix.sanitizer_flag }} -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_TOOLCHAIN_FILE=${{github.workspace}}/cross-compile/freedv-mingw-llvm-${{ matrix.arch }}.cmake ..
- name: Build freedv-gui (unsigned, ${{ matrix.sanitizer_name }})
shell: bash
working-directory: ${{github.workspace}}/build_windows
run: |
export PATH=${{github.workspace}}/llvm-mingw-20251216-ucrt-ubuntu-22.04-x86_64/bin:$PATH
make -j$(nproc) package
- name: Calculate installer filename
shell: bash
id: installer-filename
working-directory: ${{github.workspace}}/build_windows
run: |
export FREEDV_INSTALLER_FILE=`ls FreeDV*.exe`
echo "INSTALLER_FILENAME=${FREEDV_INSTALLER_FILE}" >> "$GITHUB_OUTPUT"
mv ${FREEDV_INSTALLER_FILE} sanitizer-${{ matrix.sanitizer_name }}-${FREEDV_INSTALLER_FILE}
- name: Stash for test-sanitizer job
uses: actions/upload-artifact@v7
with:
name: sanitizer-${{ matrix.sanitizer_name }}-${{ steps.installer-filename.outputs.INSTALLER_FILENAME }}
path: ${{github.workspace}}/build_windows/sanitizer-${{ matrix.sanitizer_name }}-${{ steps.installer-filename.outputs.INSTALLER_FILENAME }}
archive: false
# rade_tx_wav / rade_rx_wav are excluded from the Windows build in the rade_c
# fork (if(NOT WIN32) in src/CMakeLists.txt), so the RADE loss threshold can't be
# derived on the Windows runners. Compute the software-only baseline here on Linux
# and hand the resulting threshold to the Windows RADE loss test. See
# https://github.com/drowe67/radae/blob/dr-tx-bpf/doc/verification/verification_procedure.md
rade-loss-baseline:
runs-on: ubuntu-22.04
outputs:
baseline: ${{ steps.baseline.outputs.baseline }}
threshold: ${{ steps.baseline.outputs.threshold }}
rade_c_commit: ${{ steps.baseline.outputs.rade_c_commit }}
radae_commit: ${{ steps.baseline.outputs.radae_commit }}
steps:
- uses: actions/checkout@v6
- name: Install required packages
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y cmake autoconf automake libtool libltdl-dev pkg-config sox
- name: Build rade_c wav tools
shell: bash
working-directory: ${{github.workspace}}
run: |
set -eux
git clone -b dr-tx-bpf --depth 1 https://github.com/freedv/rade_c rade_c
cmake -S rade_c -B rade_c/build -DCMAKE_BUILD_TYPE=Release \
-DOPUS_URL=https://github.com/xiph/opus/archive/940d4e5af64351ca8ba8390df3f555484c567fbb.zip
cmake --build rade_c/build -j$(nproc)
- name: Compute RADE loss threshold
id: baseline
shell: bash
working-directory: ${{github.workspace}}
run: |
set -eux
git clone -b main --depth 1 https://github.com/drowe67/radae.git rade_src
python3 -m venv rade-venv
. ./rade-venv/bin/activate
pip3 install torch --index-url https://download.pytorch.org/whl/cpu
pip3 install matplotlib
TOOLS="$PWD/rade_c/build/src"
# rade_tx_wav requires 16 kHz mono 16-bit PCM (all.wav already is).
sox rade_src/wav/all.wav -r 16000 -c 1 -b 16 -e signed-integer baseline_in.wav
LD_LIBRARY_PATH="$TOOLS" "$TOOLS/rade_tx_wav" -f baseline_txfeatures.f32 baseline_in.wav baseline_tx.wav
LD_LIBRARY_PATH="$TOOLS" "$TOOLS/rade_rx_wav" -f baseline_rxfeatures.f32 baseline_tx.wav baseline_decoded.wav
BASELINE_OUTPUT=$(PYTHONPATH="$PWD/rade_src" python3 rade_src/loss.py \
baseline_txfeatures.f32 baseline_rxfeatures.f32 --clip_start 100 --clip_end 300)
echo "$BASELINE_OUTPUT"
BASELINE_LOSS=$(printf '%s\n' "$BASELINE_OUTPUT" | sed -n 's/.*loss: *\([0-9][0-9.]*\).*/\1/p' | head -1)
# Verification procedure: allow the round trip to be within +10% of baseline.
THRESHOLD=$(awk -v b="$BASELINE_LOSS" 'BEGIN { if (b + 0 <= 0) exit 1; printf "%.4f\n", b * 1.10 }')
echo "Software-only baseline loss: $BASELINE_LOSS -> threshold (x1.10): $THRESHOLD"
echo "baseline=$BASELINE_LOSS" >> "$GITHUB_OUTPUT"
echo "threshold=$THRESHOLD" >> "$GITHUB_OUTPUT"
echo "rade_c_commit=$(git -C rade_c rev-parse HEAD)" >> "$GITHUB_OUTPUT"
echo "radae_commit=$(git -C rade_src rev-parse HEAD)" >> "$GITHUB_OUTPUT"
collect-pgo-profile:
strategy:
fail-fast: false
@ -107,7 +248,7 @@ jobs:
- uses: actions/download-artifact@v8
with:
merge-multiple: true
pattern: instrumented-FreeDV-2*-${{ matrix.arch }}.exe
pattern: instrumented-FreeDV-*-${{ matrix.arch }}.exe
path: ${{github.workspace}}
- uses: TheMrMilchmann/setup-msvc-dev@v4
@ -380,7 +521,7 @@ jobs:
arch: x86_64
runs-on: ${{ matrix.runner }}
needs: build-pgo-use-profile
needs: [build-pgo-use-profile, rade-loss-baseline]
env:
RADIO_TO_COMPUTER_DEVICE: "CABLE Output (VB-Audio Virtual Cable)"
COMPUTER_TO_RADIO_DEVICE: "Speakers (VB-Audio Virtual Cable)"
@ -392,7 +533,7 @@ jobs:
- uses: actions/download-artifact@v8
with:
merge-multiple: true
pattern: FreeDV-2*-${{ matrix.arch }}.exe
pattern: FreeDV-*-${{ matrix.arch }}.exe
path: ${{github.workspace}}
- uses: TheMrMilchmann/setup-msvc-dev@v4
@ -438,12 +579,20 @@ jobs:
Copy-Item -Path ${{github.workspace}}/test/freedv-ctest-fullduplex.conf.tmpl -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/TestFreeDVReporting.ps1 -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/freedv-ctest-reporting.conf.tmpl -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/TestFreeDVRadeLoss.ps1 -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/freedv-ctest-loss.conf.tmpl -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/hamlibserver.py -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
- name: Install SoX
shell: pwsh
run: |
choco install sox.portable
- name: Install Python dependencies
shell: pwsh
run: |
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install matplotlib
- name: Grant FreeDV access to the microphone
run: |
@ -456,29 +605,26 @@ jobs:
New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\" -Name "AppPrivacy" -Force
Set-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\AppPrivacy" -Name LetAppsAccessMicrophone -Value 0
- name: Start Windows Audio Service
- name: Wait for virtual audio endpoints
shell: pwsh
run: |
Restart-Service -Name audiosrv -Force
$deadline = (Get-Date).AddSeconds(60)
do {
Start-Sleep -Seconds 2
$deviceNames = (Get-CimInstance win32_sounddevice).Name
} until (
($deviceNames -match "VB-Audio Virtual Cable") -and
($deviceNames -match "Virtual Audio Cable") -or
((Get-Date) -gt $deadline)
)
# Restart the audio stack and poll until every endpoint FreeDV needs is
# actually enumerable, instead of starting a test and letting it abort
# with a fatal "device cannot be found" message box when a virtual
# cable endpoint hasn't come up (or has briefly dropped) yet.
& "${{github.workspace}}\ci\Wait-AudioDevices.ps1" `
-Playback "${{ env.COMPUTER_TO_RADIO_DEVICE }}", "${{ env.COMPUTER_TO_SPEAKER_DEVICE }}" `
-Recording "${{ env.RADIO_TO_COMPUTER_DEVICE }}", "${{ env.MICROPHONE_TO_COMPUTER_DEVICE }}" `
-TimeoutSeconds 180
- name: List audio devices
shell: pwsh
run: |
Get-CimInstance win32_sounddevice
Get-AudioDevice -List | Format-Table -AutoSize Index, Default, Type, Name
- name: Set Windows default audio devices
shell: pwsh
run: |
Invoke-WebRequest https://github.com/frgnca/AudioDeviceCmdlets/releases/download/v3.1/AudioDeviceCmdlets.dll -OutFile AudioDeviceCmdlets.dll
Install-Module -Name AudioDeviceCmdlets -Force -Confirm:$false
Get-AudioDevice -List | where { $_.Name -eq "${{ env.MICROPHONE_TO_COMPUTER_DEVICE }}" } | Set-AudioDevice
Get-AudioDevice -List | where { $_.Name -eq "${{ env.COMPUTER_TO_SPEAKER_DEVICE }}" } | Set-AudioDevice
@ -494,6 +640,22 @@ jobs:
powercfg /setacvalueindex SCHEME_CURRENT 54533251-82be-4824-96c1-47b60b740d00 893dee8e-2bef-41e0-89c6-b55d0929964c 100
powercfg /setactive SCHEME_CURRENT
- name: Configure crash dump collection for FreeDV
shell: pwsh
run: |
# Windows Error Reporting will write a dump here any time freedv.exe
# crashes (unhandled exception/access violation), even if the test
# script itself doesn't notice anything went wrong (e.g. a crash
# mid-run that a subsequent step just sees as a hang or empty output).
$dumpFolder = "${{github.workspace}}\CrashDumps"
New-Item -Path $dumpFolder -ItemType Directory -Force | Out-Null
$werKey = "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\freedv.exe"
New-Item -Path $werKey -Force | Out-Null
Set-ItemProperty -Path $werKey -Name DumpFolder -Value $dumpFolder -Type ExpandString
Set-ItemProperty -Path $werKey -Name DumpCount -Value 10 -Type DWord
Set-ItemProperty -Path $werKey -Name DumpType -Value 2 -Type DWord
- name: Test RADE
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
@ -502,6 +664,14 @@ jobs:
.\TestFreeDVFullDuplex.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" -ModeToTest RADEV1 -NumberOfRuns 1
timeout-minutes: 10
- name: Test RADE Loss
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
if: ${{ !cancelled() }}
run: |
.\TestFreeDVRadeLoss.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" -LossThreshold ${{ needs.rade-loss-baseline.outputs.threshold }} *>&1 | Tee-Object -FilePath rade_loss.log
timeout-minutes: 10
- name: Test RADE Reporting
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
@ -510,30 +680,6 @@ jobs:
.\TestFreeDVReporting.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}"
timeout-minutes: 10
- name: Test 700D
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
if: ${{ !cancelled() }}
run: |
.\TestFreeDVFullDuplex.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" -ModeToTest 700D -NumberOfRuns 1
timeout-minutes: 10
- name: Test 700E
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
if: ${{ !cancelled() }}
run: |
.\TestFreeDVFullDuplex.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" -ModeToTest 700E -NumberOfRuns 1
timeout-minutes: 10
- name: Test 1600
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
if: ${{ !cancelled() }}
run: |
.\TestFreeDVFullDuplex.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" -ModeToTest 1600 -NumberOfRuns 1
timeout-minutes: 10
- name: Screenshot if failed
if: failure()
shell: pwsh
@ -549,3 +695,256 @@ jobs:
path: ${{github.workspace}}/screenshot-${{ matrix.arch }}.png
archive: false
- name: Upload crash dumps
uses: actions/upload-artifact@v7
if: always()
with:
name: crashdumps-${{ matrix.arch }}
path: ${{github.workspace}}\CrashDumps\*.dmp
if-no-files-found: ignore
# Runs the same test suite as `test`, but against the sanitizer-instrumented
# builds from `build-sanitizer`. These builds are slower and less reliable
# for real-time audio than a plain Release build (the same reason macOS/Linux
# CI treat sanitizer runs as observational rather than pass/fail-gating --
# see SANITIZERS_ENABLED in CMakeLists.txt), so the test-execution steps are
# continue-on-error: what we actually care about here is whether a sanitizer
# catches a real memory-safety/UB bug (an abort with a diagnostic in the
# log), not whether a loss threshold was met under instrumentation overhead.
test-sanitizer:
strategy:
fail-fast: false
matrix:
include:
- sanitizer_name: UBSAN
arch: x86_64
runner: windows-2022
- sanitizer_name: UBSAN
arch: aarch64
runner: windows-11-arm
- sanitizer_name: ASAN
arch: x86_64
runner: windows-2022
runs-on: ${{ matrix.runner }}
needs: [build-sanitizer, rade-loss-baseline]
env:
RADIO_TO_COMPUTER_DEVICE: "CABLE Output (VB-Audio Virtual Cable)"
COMPUTER_TO_RADIO_DEVICE: "Speakers (VB-Audio Virtual Cable)"
MICROPHONE_TO_COMPUTER_DEVICE: "Line 1 (Virtual Audio Cable)"
COMPUTER_TO_SPEAKER_DEVICE: "Line 1 (Virtual Audio Cable)"
# LSan has no Windows runtime, so it's intentionally not configured here.
# Options are colon-separated key=value pairs, so an absolute Windows
# path (drive-letter colon, e.g. D:\...) breaks the parser -- it reads
# "suppressions=D" as one pair and "\...\asan_suppressions.txt" (no
# '=') as the next, and aborts with "expected '=' in ASAN_OPTIONS"
# before FreeDV even starts. Use bare relative filenames instead,
# resolved against each test step's working directory (see the
# "Copy test scripts to install folder" step below).
ASAN_OPTIONS: suppressions=asan_suppressions.txt
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1:suppressions=ubsan_suppressions.txt
steps:
- uses: actions/checkout@v6
- uses: actions/download-artifact@v8
with:
merge-multiple: true
pattern: sanitizer-${{ matrix.sanitizer_name }}-FreeDV-*-${{ matrix.arch }}.exe
path: ${{github.workspace}}
- uses: TheMrMilchmann/setup-msvc-dev@v4
with:
arch: x64
- name: Install VAC ("analog" sound device)
run: |
Invoke-WebRequest https://software.muzychenko.net/freeware/vac470lite.zip -OutFile vac470lite.zip
Expand-Archive -Path vac470lite.zip -DestinationPath vac470lite
$driverFile = 'vac470lite\${{ env.Platform }}\vrtaucbl.sys';
$outputFile = '${{github.workspace}}\vac.cer';
$exportType = [System.Security.Cryptography.X509Certificates.X509ContentType]::Cert;
$cert = (Get-AuthenticodeSignature $driverFile).SignerCertificate;
[System.IO.File]::WriteAllBytes($outputFile, $cert.Export($exportType));
.\ci\Install-Driver.ps1 $outputFile "vac470lite/vrtaucbl.inf" VirtualAudioCable_83ed7f0e-2028-4956-b0b4-39c76fdaef1d
- name: Install VB-Cable ("Radio" sound device)
run: |
Invoke-WebRequest https://download.vb-audio.com/Download_CABLE/VBCABLE_Driver_Pack45.zip -OutFile VBCABLE_Driver_Pack45.zip
Expand-Archive -Path VBCABLE_Driver_Pack45.zip -DestinationPath VBCABLE_Driver_Pack45
$driverFile = 'VBCable_Driver_Pack45\vbaudio_cable64_win10.sys';
$outputFile = '${{github.workspace}}\vbcable.cer';
$exportType = [System.Security.Cryptography.X509Certificates.X509ContentType]::Cert;
$cert = (Get-AuthenticodeSignature $driverFile).SignerCertificate;
[System.IO.File]::WriteAllBytes($outputFile, $cert.Export($exportType));
.\ci\Install-Driver.ps1 $outputFile VBCable_Driver_Pack45\vbMmeCable64_win10.inf VBAudioVACWDM
- name: Install FreeDV on hard drive
shell: pwsh
run: |
Get-ChildItem *.exe | Rename-Item -NewName FreeDV.exe
.\FreeDV.exe /S /D=${{github.workspace}}\FreeDV-Install-Location | Out-Null
- name: Copy test scripts to install folder
shell: pwsh
run: |
Copy-Item -Path ${{github.workspace}}/test/TestFreeDVFullDuplex.ps1 -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/freedv-ctest-fullduplex.conf.tmpl -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/TestFreeDVReporting.ps1 -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/freedv-ctest-reporting.conf.tmpl -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/hamlibserver.py -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/TestFreeDVRadeLoss.ps1 -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/freedv-ctest-loss.conf.tmpl -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
# Sanitizer options need bare relative filenames (see ASAN_OPTIONS/
# UBSAN_OPTIONS above), so the suppression files need to live next
# to freedv.exe rather than being referenced by absolute path.
Copy-Item -Path ${{github.workspace}}/test/asan_suppressions.txt -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
Copy-Item -Path ${{github.workspace}}/test/ubsan_suppressions.txt -Destination ${{github.workspace}}\FreeDV-Install-Location\bin
- name: Install SoX
shell: pwsh
run: |
choco install sox.portable
- name: Install Python dependencies
shell: pwsh
run: |
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install matplotlib
- name: Grant FreeDV access to the microphone
run: |
New-Item -Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\CapabilityAccessManager\ConsentStore" -Name "microphone" -Force
Set-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\CapabilityAccessManager\ConsentStore\microphone" -Name Value -Value Allow
New-Item -Path "HKCU:\SOFTWARE\Microsoft\Windows\CurrentVersion\CapabilityAccessManager\ConsentStore" -Name "microphone" -Force
Set-ItemProperty -Path "HKCU:\SOFTWARE\Microsoft\Windows\CurrentVersion\CapabilityAccessManager\ConsentStore\microphone" -Name Value -Value Allow
New-Item -Path "HKCU:\SOFTWARE\Microsoft\Windows\CurrentVersion\CapabilityAccessManager\ConsentStore\microphone\" -Name "NonPackaged" -Force
Set-ItemProperty -Path "HKCU:\SOFTWARE\Microsoft\Windows\CurrentVersion\CapabilityAccessManager\ConsentStore\microphone\NonPackaged" -Name Value -Value Allow
New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\" -Name "AppPrivacy" -Force
Set-ItemProperty -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\AppPrivacy" -Name LetAppsAccessMicrophone -Value 0
- name: Wait for virtual audio endpoints
shell: pwsh
run: |
& "${{github.workspace}}\ci\Wait-AudioDevices.ps1" `
-Playback "${{ env.COMPUTER_TO_RADIO_DEVICE }}", "${{ env.COMPUTER_TO_SPEAKER_DEVICE }}" `
-Recording "${{ env.RADIO_TO_COMPUTER_DEVICE }}", "${{ env.MICROPHONE_TO_COMPUTER_DEVICE }}" `
-TimeoutSeconds 180
- name: List audio devices
shell: pwsh
run: |
Get-AudioDevice -List | Format-Table -AutoSize Index, Default, Type, Name
- name: Set Windows default audio devices
shell: pwsh
run: |
Get-AudioDevice -List | where { $_.Name -eq "${{ env.MICROPHONE_TO_COMPUTER_DEVICE }}" } | Set-AudioDevice
Get-AudioDevice -List | where { $_.Name -eq "${{ env.COMPUTER_TO_SPEAKER_DEVICE }}" } | Set-AudioDevice
- name: Set High Performance power plan to prevent CPU frequency scaling
if: ${{ matrix.runner == 'windows-11-arm' }}
shell: pwsh
run: |
powercfg /setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c
powercfg /setacvalueindex SCHEME_CURRENT 54533251-82be-4824-96c1-47b60b740d00 893dee8e-2bef-41e0-89c6-b55d0929964c 100
powercfg /setactive SCHEME_CURRENT
- name: Configure crash dump collection for FreeDV
shell: pwsh
run: |
$dumpFolder = "${{github.workspace}}\CrashDumps"
New-Item -Path $dumpFolder -ItemType Directory -Force | Out-Null
$werKey = "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\freedv.exe"
New-Item -Path $werKey -Force | Out-Null
Set-ItemProperty -Path $werKey -Name DumpFolder -Value $dumpFolder -Type ExpandString
Set-ItemProperty -Path $werKey -Name DumpCount -Value 10 -Type DWord
Set-ItemProperty -Path $werKey -Name DumpType -Value 2 -Type DWord
- name: Test RADE
id: test-rade
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
if: ${{ !cancelled() }}
continue-on-error: true
run: |
.\TestFreeDVFullDuplex.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" -ModeToTest RADEV1 -NumberOfRuns 1 *>&1 | Tee-Object -FilePath fullduplex.log
timeout-minutes: 15
- name: Test RADE Loss
id: test-rade-loss
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
if: ${{ !cancelled() }}
continue-on-error: true
run: |
.\TestFreeDVRadeLoss.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" -LossThreshold ${{ needs.rade-loss-baseline.outputs.threshold }} *>&1 | Tee-Object -FilePath rade_loss.log
timeout-minutes: 15
- name: Test RADE Reporting
id: test-rade-reporting
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
if: ${{ !cancelled() }}
continue-on-error: true
run: |
.\TestFreeDVReporting.ps1 -RadioToComputerDevice "${{env.RADIO_TO_COMPUTER_DEVICE}}" -ComputerToRadioDevice "${{env.COMPUTER_TO_RADIO_DEVICE}}" -MicrophoneToComputerDevice "${{env.MICROPHONE_TO_COMPUTER_DEVICE}}" -ComputerToSpeakerDevice "${{env.COMPUTER_TO_SPEAKER_DEVICE}}" *>&1 | Tee-Object -FilePath reporting.log
timeout-minutes: 15
# The three test steps above are continue-on-error so an ordinary loss-
# threshold miss (expected under instrumentation overhead, same leniency
# ctest gives sanitizer builds on macOS/Linux via SANITIZERS_ENABLED)
# doesn't stop the other tests from running or fail the job by itself.
# But that also means a *real* sanitizer abort would otherwise pass
# silently, since continue-on-error suppresses the job's conclusion, not
# just the step's outcome -- so explicitly scan every test's captured
# output for a sanitizer report and fail for real if one is found.
- name: Check for sanitizer errors
id: check-sanitizer-errors
if: ${{ !cancelled() }}
shell: pwsh
working-directory: ${{github.workspace}}\FreeDV-Install-Location\bin
run: |
$pattern = 'AddressSanitizer|UndefinedBehaviorSanitizer|LeakSanitizer'
$hits = Get-ChildItem -Path . -Filter *.log | ForEach-Object { Select-String -Path $_.FullName -Pattern $pattern }
if ($hits) {
Write-Host "::error::Sanitizer violation(s) found in test output:"
$hits | ForEach-Object { Write-Host "$($_.Filename):$($_.LineNumber): $($_.Line)" }
exit 1
}
Write-Host "No sanitizer violations found in test output."
- name: Screenshot if failed
if: ${{ steps.test-rade.outcome == 'failure' || steps.test-rade-loss.outcome == 'failure' || steps.test-rade-reporting.outcome == 'failure' || steps.check-sanitizer-errors.outcome == 'failure' }}
shell: pwsh
run: |
Invoke-WebRequest https://raw.githubusercontent.com/npocmaka/batch.scripts/refs/heads/master/hybrids/.net/c/screenCapture.bat -OutFile screenCapture.bat
& .\screenCapture.bat screenshot-${{ matrix.sanitizer_name }}-${{ matrix.arch }}.png
- name: Upload screenshot
uses: actions/upload-artifact@v7
if: ${{ steps.test-rade.outcome == 'failure' || steps.test-rade-loss.outcome == 'failure' || steps.test-rade-reporting.outcome == 'failure' || steps.check-sanitizer-errors.outcome == 'failure' }}
with:
name: screenshot-${{ matrix.sanitizer_name }}-${{ matrix.arch }}.png
path: ${{github.workspace}}/screenshot-${{ matrix.sanitizer_name }}-${{ matrix.arch }}.png
archive: false
- name: Upload crash dumps
uses: actions/upload-artifact@v7
if: always()
with:
name: crashdumps-${{ matrix.sanitizer_name }}-${{ matrix.arch }}
path: ${{github.workspace}}\CrashDumps\*.dmp
if-no-files-found: ignore
- name: Upload test logs
uses: actions/upload-artifact@v7
if: always()
with:
name: sanitizer-logs-${{ matrix.sanitizer_name }}-${{ matrix.arch }}
path: ${{github.workspace}}\FreeDV-Install-Location\bin\*.log
if-no-files-found: ignore

398
test/TestFreeDVRadeLoss.ps1 100644
View File

@ -0,0 +1,398 @@
<#
.SYNOPSIS
Executes RADE loss test of FreeDV.
.DESCRIPTION
This script pushes a known test file through FreeDV's TX pipeline while recording the resulting audio from
the real Windows sound card path, then plays that recording back into FreeDV's RX pipeline. The RADE features
captured on the TX and RX sides are compared using the loss.py tool from the radae repository, which prints
PASS or FAIL depending on whether the feature loss introduced by the round trip through real audio hardware
is below the given threshold. This is the PowerShell equivalent of test/test_rade_loss.sh.
When -LossThreshold is not supplied, the threshold is derived from a software-only baseline as described in
the RADE integration verification procedure
(https://github.com/drowe67/radae/blob/dr-tx-bpf/doc/verification/verification_procedure.md): all.wav is run
through rade_tx_wav + rade_rx_wav and loss.py, and the hardware round trip must stay within +10% of that
baseline loss. rade_tx_wav / rade_rx_wav are not built for Windows in the rade_c fork, so Windows CI computes
the baseline on a Linux runner and passes the value in via -LossThreshold; -RadeCToolsDir lets a local run
point at a rade_c build that does have the tools.
.INPUTS
None. You can't pipe objects to this script.
.OUTPUTS
The script outputs FreeDV and loss.py logging as well as a final Passed/Failures summary to the console.
.EXAMPLE
PS> .\TestFreeDVRadeLoss.ps1 -RadioToComputerDevice "CABLE Output (VB-Audio Virtual Cable)" -ComputerToRadioDevice "Speakers (VB-Audio Virtual Cable)" -ComputerToSpeakerDevice "Line 1 (Virtual Audio Cable)" -MicrophoneToComputerDevice "Line 1 (Virtual Audio Cable)"
#>
param (
[Parameter(Mandatory = $true)]
[ValidateNotNullOrEmpty()]
[string]
# The sound device to receive RX audio from. Also used to record FreeDV's TX output, as it is the
# recording half of the same virtual cable that ComputerToRadioDevice plays into.
$RadioToComputerDevice,
[Parameter(Mandatory = $true)]
[ValidateNotNullOrEmpty()]
[string]
# The sound device to emit decoded audio to.
$ComputerToSpeakerDevice,
[Parameter(Mandatory = $true)]
[ValidateNotNullOrEmpty()]
[string]
# The sound device to receive analog audio from.
$MicrophoneToComputerDevice,
[Parameter(Mandatory = $true)]
[ValidateNotNullOrEmpty()]
[string]
# The sound device to emit TX audio to. Also used to play the recorded TX audio back into
# FreeDV's RX input, as it is the playback half of the same virtual cable that RadioToComputerDevice records from.
$ComputerToRadioDevice,
[double]
# The maximum acceptable RADE feature loss fraction before the test is considered failed.
# When left at 0 (the default) it is computed from a software-only baseline (see .DESCRIPTION):
# baseline loss from rade_tx_wav/rade_rx_wav on all.wav, times 1.10.
$LossThreshold = 0,
[string]
# Directory containing rade_tx_wav.exe / rade_rx_wav.exe (from a rade_c build), used to compute
# the baseline when -LossThreshold is not supplied. Auto-detected when possible.
$RadeCToolsDir = "",
[string]
# Path or filename of the Python interpreter used to run loss.py.
$PythonBinary = "python.exe")
# Fallback threshold used only when -LossThreshold is not supplied and the software-only
# baseline cannot be computed (e.g. rade_tx_wav / rade_rx_wav unavailable).
$FallbackLossThreshold = 0.0891
$LossTolerance = 1.10
<#
.Description
Runs all.wav through the rade_c software-only path (rade_tx_wav -> rade_rx_wav -> loss.py) and returns
the baseline feature loss multiplied by $LossTolerance, i.e. the maximum loss the hardware round trip is
allowed to introduce. Returns $null if the tools, test corpus or loss figure can't be found, in which
case the caller should fall back to $FallbackLossThreshold. Mirrors compute_loss_threshold() in
test/test_rade_loss.sh.
#>
function Get-RadeLossThreshold {
param (
$current_loc,
$RadeCToolsDir,
$PythonBinary,
$Tolerance
)
# Locate rade_tx_wav.exe / rade_rx_wav.exe from a rade_c build.
$candidates = New-Object System.Collections.Generic.List[string]
if ($RadeCToolsDir) { $candidates.Add($RadeCToolsDir) }
$candidates.Add("$current_loc")
$candidates.Add((Join-Path $current_loc "_deps\freedv_backend-build\rade_build\src"))
$found = Get-ChildItem -Path $current_loc -Recurse -Filter "rade_tx_wav.exe" -ErrorAction SilentlyContinue | Select-Object -First 1
if ($found) { $candidates.Add($found.DirectoryName) }
$toolsDir = $null
foreach ($c in $candidates) {
if ($c -and (Test-Path (Join-Path $c "rade_tx_wav.exe")) -and (Test-Path (Join-Path $c "rade_rx_wav.exe"))) {
$toolsDir = $c
break
}
}
if (-not $toolsDir) {
Write-Host "Could not find rade_tx_wav.exe / rade_rx_wav.exe; pass -RadeCToolsDir or -LossThreshold."
return $null
}
$allWav = Join-Path $current_loc "rade_src\wav\all.wav"
if (-not (Test-Path $allWav)) {
Write-Host "Could not find $allWav for baseline computation."
return $null
}
$baseIn = Join-Path $current_loc "baseline_in.wav"
$baseTxWav = Join-Path $current_loc "baseline_tx.wav"
$baseDecoded = Join-Path $current_loc "baseline_decoded.wav"
$baseTxF = Join-Path $current_loc "baseline_txfeatures.f32"
$baseRxF = Join-Path $current_loc "baseline_rxfeatures.f32"
# rade_tx_wav requires 16 kHz mono 16-bit PCM (all.wav already is); normalise defensively.
# Output of the native tools is captured (not left on the pipeline) so it can't corrupt the return value.
$toolOut = & sox.exe $allWav -r 16000 -c 1 -b 16 -e signed-integer $baseIn 2>&1
if ($LASTEXITCODE -ne 0) { Write-Host "$toolOut"; return $null }
# Prepend the tools directory to PATH so librade.dll resolves next to the executables.
$oldPath = $env:PATH
$env:PATH = "$toolsDir;$env:PATH"
try {
# --v2 to match the mode FreeDV runs: src/freedv_interface.cpp opens RADE
# with RADE_MODE_V2 unconditionally on this branch, whatever the -utmode label.
$toolOut = & (Join-Path $toolsDir "rade_tx_wav.exe") --v2 -f $baseTxF $baseIn $baseTxWav 2>&1
if ($LASTEXITCODE -ne 0) { Write-Host "$toolOut"; return $null }
$toolOut = & (Join-Path $toolsDir "rade_rx_wav.exe") --v2 -f $baseRxF $baseTxWav $baseDecoded 2>&1
if ($LASTEXITCODE -ne 0) { Write-Host "$toolOut"; return $null }
}
finally {
$env:PATH = $oldPath
}
$baselineOutput = & $PythonBinary (Join-Path $current_loc "rade_src\loss.py") $baseTxF $baseRxF --clip_start 100 --clip_end 300 2>&1
Write-Host "software-only baseline: $baselineOutput"
$m = [regex]::Match(($baselineOutput -join "`n"), 'loss:\s*([0-9]+\.?[0-9]*)')
if (-not $m.Success) { return $null }
$baselineLoss = [double]$m.Groups[1].Value
if ($baselineLoss -le 0) { return $null }
return [math]::Round($baselineLoss * $Tolerance, 4)
}
<#
.Description
Starts FreeDV in RX mode listening on the real sound card, plays the given wav file back into it, then
compares the resulting RX RADE features against the reference txfeatures.f32 using loss.py. Returns
$true/$false depending on whether the loss is within the given threshold. Factored out of Test-RadeLoss
so it can be retried against a phase-shifted copy of the recording (see the RADEV2 sync-timing workaround
below).
#>
function Invoke-RadeLossAttempt {
param (
$current_loc,
$psi,
$ComputerToRadioDevice,
$PlaybackFile,
$PythonBinary,
$LossThreshold
)
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
$process.PriorityClass = [System.Diagnostics.ProcessPriorityClass]::AboveNormal
Start-Sleep -Milliseconds 4995
# Play the recorded TX audio back into FreeDV's RX input
$playPsi = New-Object System.Diagnostics.ProcessStartInfo
$playPsi.CreateNoWindow = $true
$playPsi.UseShellExecute = $false
$playPsi.RedirectStandardError = $false
$playPsi.RedirectStandardOutput = $false
$playPsi.FileName = "sox.exe"
$playPsi.WorkingDirectory = $current_loc
$quoted_play_device = "`"" + $ComputerToRadioDevice + "`""
$playPsi.Arguments = @("--buffer 128000 -t wav `"$PlaybackFile`" -t waveaudio $quoted_play_device")
$playProcess = New-Object System.Diagnostics.Process
$playProcess.StartInfo = $playPsi
[void]$playProcess.Start()
# Read output from RX run
$err_output = $process.StandardError.ReadToEnd();
$output = $process.StandardOutput.ReadToEnd();
$process.WaitForExit()
Write-Host "$err_output"
$rxExitCode = $process.ExitCode
try {
if (-not $playProcess.HasExited) {
$playProcess.Kill()
}
} catch {
# Ignore failure as the process may have already exited
}
$playProcess.WaitForExit()
if ($rxExitCode -ne 0) {
return $false
}
# Compare TX/RX RADE features using loss.py from the radae repo
$lossPsi = New-Object System.Diagnostics.ProcessStartInfo
$lossPsi.CreateNoWindow = $true
$lossPsi.UseShellExecute = $false
$lossPsi.RedirectStandardError = $true
$lossPsi.RedirectStandardOutput = $true
$lossPsi.FileName = $PythonBinary
$lossPsi.WorkingDirectory = $current_loc
$lossPsi.Arguments = @("`"$current_loc\rade_src\loss.py`" `"$current_loc\txfeatures.f32`" `"$current_loc\rxfeatures.f32`" --loss_test $LossThreshold --clip_start 100 --clip_end 300")
$lossProcess = New-Object System.Diagnostics.Process
$lossProcess.StartInfo = $lossPsi
[void]$lossProcess.Start()
$loss_output = $lossProcess.StandardOutput.ReadToEnd();
$loss_err = $lossProcess.StandardError.ReadToEnd();
$lossProcess.WaitForExit()
Write-Host "$loss_output"
Write-Host "$loss_err"
$lossPasses = ($loss_output -split "`r?`n") | Where { $_.Contains("PASS") }
return (($lossProcess.ExitCode -eq 0) -and ($lossPasses.Count -ge 1))
}
<#
.Description
Performs the actual test by cloning the radae test corpus, transmitting a known test file through FreeDV,
recording the resulting audio, playing it back through FreeDV's RX pipeline, and comparing the TX/RX RADE
features using loss.py.
#>
function Test-RadeLoss {
param (
$RadioToComputerDevice,
$ComputerToSpeakerDevice,
$MicrophoneToComputerDevice,
$ComputerToRadioDevice,
$LossThreshold,
$RadeCToolsDir,
$PythonBinary
)
$current_loc = Get-Location
# Clone radae repo (contains the test audio and the loss.py comparison tool) if not already present.
if (-not (Test-Path "$current_loc\rade_src")) {
& git.exe clone -b main https://github.com/drowe67/radae.git "$current_loc\rade_src"
}
# Resample test file to 48 kHz. Needed for CI environment to reduce CPU usage.
& sox.exe "$current_loc\rade_src\wav\all.wav" -r 48000 "$current_loc\tx_in.wav"
# Resolve the loss threshold: use -LossThreshold when supplied, otherwise derive it from the
# software-only baseline (see .DESCRIPTION), falling back to a fixed value if that can't be done.
if ($LossThreshold -le 0) {
$computed = Get-RadeLossThreshold -current_loc $current_loc -RadeCToolsDir $RadeCToolsDir -PythonBinary $PythonBinary -Tolerance $LossTolerance
if ($null -ne $computed) {
$LossThreshold = $computed
Write-Host "RADE loss threshold: $LossThreshold (software-only baseline x $LossTolerance)"
} else {
$LossThreshold = $FallbackLossThreshold
Write-Host "WARNING: could not compute RADE loss baseline; using fallback threshold $LossThreshold"
}
} else {
Write-Host "RADE loss threshold: $LossThreshold (supplied via -LossThreshold)"
}
# Generate new conf
$conf_tmpl = Get-Content "$current_loc\freedv-ctest-loss.conf.tmpl"
$conf_tmpl = $conf_tmpl.Replace("@FREEDV_RADIO_TO_COMPUTER_DEVICE@", $RadioToComputerDevice)
$conf_tmpl = $conf_tmpl.Replace("@FREEDV_COMPUTER_TO_RADIO_DEVICE@", $ComputerToRadioDevice)
$conf_tmpl = $conf_tmpl.Replace("@FREEDV_MICROPHONE_TO_COMPUTER_DEVICE@", $MicrophoneToComputerDevice)
$conf_tmpl = $conf_tmpl.Replace("@FREEDV_COMPUTER_TO_SPEAKER_DEVICE@", $ComputerToSpeakerDevice)
$tmp_file = New-TemporaryFile
$conf_tmpl | Set-Content -Path $tmp_file.FullName
$quoted_tmp_filename = "`"" + $tmp_file.FullName + "`""
# Start recording FreeDV's TX output
$recordPsi = New-Object System.Diagnostics.ProcessStartInfo
$recordPsi.CreateNoWindow = $true
$recordPsi.UseShellExecute = $false
$recordPsi.RedirectStandardError = $false
$recordPsi.RedirectStandardOutput = $false
$recordPsi.FileName = "sox.exe"
$recordPsi.WorkingDirectory = $current_loc
$quoted_record_device = "`"" + $RadioToComputerDevice + "`""
$recordPsi.Arguments = @("--buffer 128000 -t waveaudio $quoted_record_device -c 1 -t wav -r 48000 -b 16 -e signed-integer `"$current_loc\test.wav`"")
$recordProcess = New-Object System.Diagnostics.Process
$recordProcess.StartInfo = $recordPsi
[void]$recordProcess.Start()
# Start freedv.exe in TX mode
$psi = New-Object System.Diagnostics.ProcessStartInfo
$psi.CreateNoWindow = $true
$psi.UseShellExecute = $false
$psi.RedirectStandardError = $true
$psi.RedirectStandardOutput = $true
$psi.FileName = "$current_loc\freedv.exe"
$psi.WorkingDirectory = $current_loc
$psi.Arguments = @("/f $quoted_tmp_filename /ut tx /utmode RADEV1 /txfile `"$current_loc\tx_in.wav`" /txfeaturefile `"$current_loc\txfeatures.f32`"")
$process = New-Object System.Diagnostics.Process
$process.StartInfo = $psi
[void]$process.Start()
$process.PriorityClass = [System.Diagnostics.ProcessPriorityClass]::AboveNormal
# Read output from TX run
$err_output = $process.StandardError.ReadToEnd();
$output = $process.StandardOutput.ReadToEnd();
$process.WaitForExit()
Write-Host "$err_output"
$txExitCode = $process.ExitCode
# Stop recording
try {
$recordProcess.Kill()
} catch {
# Ignore failure as the process may have already exited
}
$recordProcess.WaitForExit()
if ($txExitCode -ne 0) {
return $false
}
# Workaround/performance improvement: strip silence at beginning and end of recording
# As well as reducing the amount of audio that needs to be played back, it also helps
# ensure we don't accidentally run into a potential RADEV2 bug (https://github.com/freedv/rade_c/issues/8)
# Note: commands adapted from https://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/
$recordPsi.Arguments = @("test.wav test_stripped.wav silence 1 0.1 1% reverse")
$stripProcess = New-Object System.Diagnostics.Process
$stripProcess.StartInfo = $recordPsi
[void]$stripProcess.Start()
$stripProcess.WaitForExit()
$recordPsi.Arguments = @("test_stripped.wav test.wav silence 1 0.1 1% reverse")
$stripProcess = New-Object System.Diagnostics.Process
$stripProcess.StartInfo = $recordPsi
[void]$stripProcess.Start()
$stripProcess.WaitForExit()
# Restart FreeDV in RX mode, reading live from the sound card so that any dropouts introduced by the
# real audio path get captured in the RX feature file (mirrors test/test_rade_loss.sh).
$psi.Arguments = @("/f $quoted_tmp_filename /ut rx /utmode RADEV1 /txtime 70 /rxfeaturefile `"$current_loc\rxfeatures.f32`"")
$passed = Invoke-RadeLossAttempt -current_loc $current_loc -psi $psi -ComputerToRadioDevice $ComputerToRadioDevice -PlaybackFile "$current_loc\test.wav" -PythonBinary $PythonBinary -LossThreshold $LossThreshold
return $passed
}
$passes = 0
$fails = 0
$result = Test-RadeLoss `
-RadioToComputerDevice $RadioToComputerDevice `
-ComputerToSpeakerDevice $ComputerToSpeakerDevice `
-MicrophoneToComputerDevice $MicrophoneToComputerDevice `
-ComputerToRadioDevice $ComputerToRadioDevice `
-LossThreshold $LossThreshold `
-RadeCToolsDir $RadeCToolsDir `
-PythonBinary $PythonBinary
if ($result -eq $true)
{
$passes++
}
else
{
$fails++
}
Write-Host "Mode: RADEV1, Passed: $passes, Failures: $fails"
if ($fails -gt 0) {
throw "Test failed"
exit 1
}

2
test/ubsan_suppressions.txt
View File

@ -3,3 +3,5 @@ vptr:wx/window.h
vptr:src/osx/core/colour.cpp
vptr:src/osx/window_osx.cpp
vptr:src/common/containr.cpp
null:src/msw/tooltip.cpp
function:crypto/stack/stack.c