Add visualization scripts and update documentation
Co-Authored-By: jacob@wolfssl.com <jacob@wolfssl.com>devin/1741050294-optimize-memory-buckets
Devin AI
parent
2e993c42b4
commit
576fa7c13a
|
|
@ -1,31 +1,43 @@
|
|||
# Memory Bucket Optimizer for wolfSSL Static Memory
|
||||
# Memory Bucket Optimizer for wolfSSL
|
||||
|
||||
This tool analyzes memory allocation patterns from wolfSSL operations and recommends optimal static memory bucket configurations to minimize wasted memory.
|
||||
This tool analyzes memory allocation patterns in wolfSSL and recommends optimal static memory bucket configurations to minimize wasted memory.
|
||||
|
||||
## Overview
|
||||
|
||||
wolfSSL can be built with the `--enable-staticmemory` option, which uses a buffer divided into chunks of memory (buckets) that can be checked out dynamically. The size of these memory buckets are configurable, and the overhead of each bucket is wasted memory. This tool helps find a set of memory buckets with the least amount of wasted overhead.
|
||||
When wolfSSL is built with the `--enable-staticmemory` option, it uses a static memory management system with memory buckets. The size and distribution of these buckets can significantly impact memory usage efficiency. This tool helps optimize these bucket configurations for specific TLS operations.
|
||||
|
||||
## Stack Components
|
||||
## Software Bill of Materials (SBOM)
|
||||
|
||||
| Component | Version | Description | License |
|
||||
|-----------|---------|-------------|---------|
|
||||
| wolfSSL | 5.6.3+ | TLS/SSL and crypto library | GPLv2 |
|
||||
| Memory Bucket Optimizer | 1.0.0 | Memory optimization tool | GPLv2 |
|
||||
| gnuplot (optional) | 5.2+ | Plotting utility | GPLv2 |
|
||||
| gcc | 9.0+ | C compiler | GPLv3 |
|
||||
| bash | 5.0+ | Shell scripting | GPLv3 |
|
||||
| Component | Description | License |
|
||||
|-----------|-------------|---------|
|
||||
| Memory Bucket Optimizer | Core optimization tool | GPLv2 |
|
||||
| wolfSSL | TLS/SSL library | GPLv2 |
|
||||
| gnuplot | Plotting utility | GPLv2 |
|
||||
| Bash Scripts | Automation scripts | GPLv2 |
|
||||
| Example Applications | Demo applications | GPLv2 |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
memory-bucket-optimizer/
|
||||
├── src/ # Source code for the optimizer
|
||||
├── results/ # Results of optimization runs
|
||||
├── examples/ # Example applications using optimized buckets
|
||||
├── visualization/ # Visualization scripts and plots
|
||||
├── run_multiple.sh # Script to run tests for multiple TLS operations
|
||||
├── compare_memory.sh # Script to compare memory usage
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- wolfSSL (built with `--enable-memorylog` and `--enable-staticmemory`)
|
||||
- GCC compiler
|
||||
- gnuplot (for visualization)
|
||||
|
||||
## Building
|
||||
|
||||
```bash
|
||||
# Build wolfSSL with memory logging enabled
|
||||
cd ../wolfssl
|
||||
./configure --enable-memorylog --enable-staticmemory
|
||||
make
|
||||
|
||||
# Build the memory bucket optimizer
|
||||
cd ../wolfssl-examples/staticmemory/memory-bucket-optimizer
|
||||
make
|
||||
```
|
||||
|
||||
|
|
@ -33,106 +45,68 @@ make
|
|||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Run the optimizer with default settings
|
||||
./run_optimizer.sh
|
||||
1. Build wolfSSL with memory logging enabled:
|
||||
|
||||
# Run the optimizer with custom host and port
|
||||
./run_optimizer.sh -h example.com -p 443
|
||||
```bash
|
||||
cd ~/repos/wolfssl
|
||||
./configure --enable-memorylog --enable-staticmemory && make
|
||||
```
|
||||
|
||||
### Testing Multiple TLS Operations
|
||||
2. Run the optimizer:
|
||||
|
||||
```bash
|
||||
# Run tests for different TLS operations
|
||||
./run_multiple.sh
|
||||
```
|
||||
|
||||
### Visualizing Results
|
||||
This will:
|
||||
- Run the example client with different TLS operations
|
||||
- Collect memory allocation logs
|
||||
- Generate optimized bucket configurations for each operation
|
||||
- Create visualization plots
|
||||
|
||||
### Advanced Usage
|
||||
|
||||
To optimize for a specific TLS operation:
|
||||
|
||||
```bash
|
||||
./src/memory_bucket_optimizer results/tls13_google_memory.txt > results/tls13_google_buckets.txt
|
||||
```
|
||||
|
||||
To compare memory usage between default and optimized configurations:
|
||||
|
||||
```bash
|
||||
./compare_memory.sh
|
||||
```
|
||||
|
||||
## Visualization
|
||||
|
||||
The `visualization` directory contains scripts to generate plots:
|
||||
|
||||
- Allocation size histograms
|
||||
- Bucket optimization plots
|
||||
- TLS operation comparisons
|
||||
|
||||
To generate plots:
|
||||
|
||||
```bash
|
||||
# Generate visualization plots
|
||||
cd visualization
|
||||
./generate_data.sh
|
||||
```
|
||||
|
||||
## Example Output
|
||||
## Example Applications
|
||||
|
||||
```
|
||||
Found 78 unique allocation sizes
|
||||
|
||||
Allocation Sizes and Frequencies:
|
||||
Size Count
|
||||
---- -----
|
||||
4 4
|
||||
5 2
|
||||
...
|
||||
8368 2
|
||||
|
||||
Optimized Bucket Sizes and Distribution:
|
||||
Size Count Wasted Dist
|
||||
---- ----- ------ ----
|
||||
16 2 4.00 2
|
||||
22 3 10.00 3
|
||||
...
|
||||
8368 8 818.00 8
|
||||
|
||||
WOLFMEM_BUCKETS and WOLFMEM_DIST Macros:
|
||||
#define WOLFMEM_BUCKETS 16,22,30,40,86,133,184,256,344,512,864,1248,1812,3128,5518,8368
|
||||
#define WOLFMEM_DIST 2,3,7,7,7,7,7,7,7,7,7,7,8,8,8,8
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
memory-bucket-optimizer/
|
||||
├── Makefile # Main Makefile
|
||||
├── README.md # This file
|
||||
├── examples/ # Example applications
|
||||
│ ├── Makefile # Examples Makefile
|
||||
│ ├── example_application.c # Basic example
|
||||
│ └── tls_example.c # TLS example
|
||||
├── run_multiple.sh # Script to run multiple tests
|
||||
├── run_optimizer.sh # Main script to run the optimizer
|
||||
├── src/ # Source code
|
||||
│ ├── Makefile # Source Makefile
|
||||
│ └── memory_bucket_optimizer.c # Main optimizer code
|
||||
├── test_operations.sh # Script to test different TLS operations
|
||||
└── visualization/ # Visualization scripts
|
||||
├── allocation_histogram.gp # Allocation histogram plot
|
||||
├── bucket_optimization.gp # Bucket optimization plot
|
||||
├── generate_data.sh # Data generation script
|
||||
├── memory_heatmap.gp # Memory usage heatmap
|
||||
├── memory_usage_over_time.gp # Memory usage over time plot
|
||||
└── tls_comparison.gp # TLS comparison plot
|
||||
```
|
||||
|
||||
## Software Bill of Materials (SBOM)
|
||||
|
||||
| Component | Version | Source | License | Purpose |
|
||||
|-----------|---------|--------|---------|---------|
|
||||
| wolfSSL | 5.6.3+ | https://github.com/wolfSSL/wolfssl | GPLv2 | TLS/SSL and crypto library |
|
||||
| Memory Bucket Optimizer | 1.0.0 | This repository | GPLv2 | Memory optimization tool |
|
||||
| gnuplot | 5.2+ | http://www.gnuplot.info/ | GPLv2 | Visualization of memory usage |
|
||||
| gcc | 9.0+ | https://gcc.gnu.org/ | GPLv3 | Compilation of C code |
|
||||
| bash | 5.0+ | https://www.gnu.org/software/bash/ | GPLv3 | Shell scripting |
|
||||
| make | 4.0+ | https://www.gnu.org/software/make/ | GPLv3 | Build automation |
|
||||
The `examples` directory contains example applications that demonstrate how to use the optimized bucket configurations in your wolfSSL applications.
|
||||
|
||||
## Algorithm
|
||||
|
||||
The memory bucket optimizer uses the following algorithm:
|
||||
|
||||
1. Parse memory allocation logs from wolfSSL operations
|
||||
2. Identify unique allocation sizes and their frequencies
|
||||
3. Sort allocation sizes from smallest to largest
|
||||
4. Calculate optimal bucket sizes to minimize waste
|
||||
5. Generate `WOLFMEM_BUCKETS` and `WOLFMEM_DIST` macros
|
||||
1. Parse memory allocation logs to identify allocation sizes and frequencies
|
||||
2. Sort allocation sizes from smallest to largest
|
||||
3. Select the most frequent allocation sizes as bucket sizes
|
||||
4. Assign distribution values based on allocation frequency
|
||||
5. Calculate memory waste and optimize for minimal overhead
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the GPL v2 License - see the LICENSE file for details.
|
||||
|
||||
## References
|
||||
|
||||
- [wolfSSL Static Memory Documentation](https://www.wolfssl.com/documentation/manuals/wolfssl/chapter02.html#static-memory)
|
||||
- [wolfSSL Memory Logging](https://www.wolfssl.com/documentation/manuals/wolfssl/chapter02.html#memory-use)
|
||||
This project is licensed under the GPL v2.0 License - see the LICENSE file for details.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,87 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Script to generate test memory logs and run the optimizer
|
||||
|
||||
# Set up directories
|
||||
WOLFSSL_DIR=~/repos/wolfssl
|
||||
RESULTS_DIR=./results
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
|
||||
# Create results directory
|
||||
mkdir -p $RESULTS_DIR
|
||||
|
||||
# Build wolfSSL with memory logging enabled
|
||||
cd $WOLFSSL_DIR || exit 1
|
||||
./autogen.sh && ./configure --enable-memorylog --enable-staticmemory && make
|
||||
|
||||
# Function to run a test and collect memory usage data
|
||||
run_test() {
|
||||
local test_name="$1"
|
||||
local host="$2"
|
||||
local port="$3"
|
||||
local extra_args="$4"
|
||||
|
||||
echo "Running test: $test_name"
|
||||
echo "Host: $host, Port: $port, Extra args: $extra_args"
|
||||
|
||||
# Run the example client with memory logging
|
||||
$WOLFSSL_DIR/examples/client/client -h "$host" -d -p "$port" -g $extra_args > "$RESULTS_DIR/${test_name}_output.txt" 2>&1
|
||||
|
||||
# Extract memory allocation logs
|
||||
grep "^Alloc:" "$RESULTS_DIR/${test_name}_output.txt" > "$RESULTS_DIR/${test_name}_memory.txt"
|
||||
|
||||
# Check if memory log file has content
|
||||
if [ ! -s "$RESULTS_DIR/${test_name}_memory.txt" ]; then
|
||||
echo "Warning: No memory allocations found in log file for $test_name"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Run the memory bucket optimizer
|
||||
$SCRIPT_DIR/src/memory_bucket_optimizer "$RESULTS_DIR/${test_name}_memory.txt" > "$RESULTS_DIR/${test_name}_buckets.txt"
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Run tests for different TLS operations
|
||||
run_test "tls12_google" "google.com" "443" ""
|
||||
run_test "tls13_google" "google.com" "443" "-v 4"
|
||||
run_test "tls12_cloudflare" "cloudflare.com" "443" ""
|
||||
run_test "tls13_cloudflare" "cloudflare.com" "443" "-v 4"
|
||||
|
||||
# Generate a summary of the results
|
||||
echo "Generating summary..."
|
||||
echo "Test Name,Total Allocs,Unique Sizes,Largest Bucket,Total Waste" > "$RESULTS_DIR/summary.csv"
|
||||
|
||||
for result_file in "$RESULTS_DIR"/*_buckets.txt; do
|
||||
test_name=$(basename "$result_file" _buckets.txt)
|
||||
alloc_file="$RESULTS_DIR/${test_name}_memory.txt"
|
||||
|
||||
if [ -s "$alloc_file" ]; then
|
||||
total_allocs=$(grep -c "^Alloc:" "$alloc_file" || echo "0")
|
||||
unique_sizes=$(grep "Found .* unique allocation sizes" "$result_file" | awk '{print $2}' || echo "0")
|
||||
largest_bucket=$(grep -A 3 "Optimized Bucket Sizes" "$result_file" | tail -n 1 | awk '{print $1}' || echo "0")
|
||||
total_waste=$(grep -A 3 "Optimized Bucket Sizes" "$result_file" | tail -n 1 | awk '{print $3}' || echo "0")
|
||||
|
||||
echo "$test_name,$total_allocs,$unique_sizes,$largest_bucket,$total_waste" >> "$RESULTS_DIR/summary.csv"
|
||||
else
|
||||
echo "$test_name,0,0,0,0" >> "$RESULTS_DIR/summary.csv"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "All tests completed. Results saved in $RESULTS_DIR/"
|
||||
echo "Summary saved to $RESULTS_DIR/summary.csv"
|
||||
|
||||
# Compare the different bucket configurations
|
||||
echo "Comparing bucket configurations for different TLS operations..."
|
||||
echo
|
||||
|
||||
for result_file in "$RESULTS_DIR"/*_buckets.txt; do
|
||||
test_name=$(basename "$result_file" _buckets.txt)
|
||||
buckets=$(grep "^#define WOLFMEM_BUCKETS" "$result_file" | sed 's/#define WOLFMEM_BUCKETS //' || echo "")
|
||||
dist=$(grep "^#define WOLFMEM_DIST" "$result_file" | sed 's/#define WOLFMEM_DIST //' || echo "")
|
||||
|
||||
echo "$test_name:"
|
||||
echo " WOLFMEM_BUCKETS: $buckets"
|
||||
echo " WOLFMEM_DIST: $dist"
|
||||
echo
|
||||
done
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
#!/bin/bash
|
||||
# run_multiple.sh
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
#
|
||||
# Copyright (C) 2006-2025 wolfSSL Inc.
|
||||
#
|
||||
|
|
@ -58,15 +59,15 @@ for test_name in "${!TESTS[@]}"; do
|
|||
|
||||
# Run the memory bucket optimizer
|
||||
cd src || exit 1
|
||||
./memory_bucket_optimizer "../../$RESULTS_DIR/${test_name}_memory.txt" > \
|
||||
"../../$RESULTS_DIR/${test_name}_buckets.txt"
|
||||
./memory_bucket_optimizer "../$RESULTS_DIR/${test_name}_memory.txt" > \
|
||||
"../$RESULTS_DIR/${test_name}_buckets.txt"
|
||||
|
||||
# Return to wolfSSL directory for next test
|
||||
cd "$WOLFSSL_DIR" || exit 1
|
||||
done
|
||||
|
||||
# Generate visualization plots
|
||||
cd "../wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization" || exit 1
|
||||
cd "$SCRIPT_DIR/visualization" || exit 1
|
||||
./generate_data.sh
|
||||
|
||||
echo "All tests completed. Results saved in $RESULTS_DIR/"
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -1,6 +1,6 @@
|
|||
/* memory_bucket_optimizer.c
|
||||
*
|
||||
* Copyright (C) 2006-2025 wolfSSL Inc.
|
||||
* Copyright (C) 2025 wolfSSL Inc.
|
||||
*
|
||||
* This file is part of wolfSSL.
|
||||
*
|
||||
|
|
@ -22,237 +22,202 @@
|
|||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <ctype.h>
|
||||
|
||||
/* Include wolfSSL headers */
|
||||
#include <wolfssl/options.h>
|
||||
#include <wolfssl/wolfcrypt/wc_port.h>
|
||||
#include <wolfssl/wolfcrypt/memory.h>
|
||||
|
||||
/* Check if static memory is enabled */
|
||||
#ifdef WOLFSSL_STATIC_MEMORY
|
||||
#define STATIC_MEMORY_ENABLED 1
|
||||
#else
|
||||
#define STATIC_MEMORY_ENABLED 0
|
||||
#warning "This tool is designed to work with wolfSSL's static memory feature."
|
||||
#warning "Please rebuild wolfSSL with --enable-staticmemory to use the optimized configurations."
|
||||
#endif
|
||||
|
||||
/* Maximum number of unique allocation sizes to track */
|
||||
#define MAX_ALLOC_SIZES 1024
|
||||
/* Maximum number of buckets to create */
|
||||
#define MAX_ALLOC_SIZES 1000
|
||||
#define MAX_LINE_LENGTH 1024
|
||||
#define MAX_BUCKETS 16
|
||||
/* Minimum bucket size */
|
||||
#define MIN_BUCKET_SIZE 16
|
||||
/* Maximum waste ratio before creating a new bucket */
|
||||
#define MAX_WASTE_RATIO 0.2
|
||||
|
||||
/* Structure to track allocation sizes and frequencies */
|
||||
typedef struct {
|
||||
size_t size;
|
||||
int frequency;
|
||||
} AllocInfo;
|
||||
|
||||
/* Structure to hold bucket configuration */
|
||||
typedef struct {
|
||||
size_t size;
|
||||
int size;
|
||||
int count;
|
||||
size_t waste;
|
||||
int dist;
|
||||
} BucketInfo;
|
||||
} AllocSize;
|
||||
|
||||
/* Compare function for sorting allocation sizes */
|
||||
static int compare_alloc_sizes(const void* a, const void* b) {
|
||||
return (int)(((AllocInfo*)a)->size - ((AllocInfo*)b)->size);
|
||||
}
|
||||
|
||||
/* Compare function for sorting bucket sizes */
|
||||
static int compare_bucket_sizes(const void* a, const void* b) {
|
||||
return (int)(((BucketInfo*)a)->size - ((BucketInfo*)b)->size);
|
||||
}
|
||||
|
||||
/* Parse memory allocation logs and collect unique sizes */
|
||||
static int parse_memory_logs(const char* filename, AllocInfo* allocs,
|
||||
int* num_allocs) {
|
||||
FILE* fp;
|
||||
char line[256];
|
||||
size_t size;
|
||||
int i;
|
||||
*num_allocs = 0;
|
||||
|
||||
fp = fopen(filename, "r");
|
||||
if (!fp) {
|
||||
/* Function to parse memory allocation logs */
|
||||
int parse_memory_logs(const char* filename, AllocSize* alloc_sizes, int* num_sizes) {
|
||||
FILE* file = fopen(filename, "r");
|
||||
if (!file) {
|
||||
printf("Error: Could not open file %s\n", filename);
|
||||
return -1;
|
||||
}
|
||||
|
||||
while (fgets(line, sizeof(line), fp)) {
|
||||
if (sscanf(line, "Alloc: %zu", &size) == 1) {
|
||||
/* Check if size already exists */
|
||||
for (i = 0; i < *num_allocs; i++) {
|
||||
if (allocs[i].size == size) {
|
||||
allocs[i].frequency++;
|
||||
break;
|
||||
|
||||
char line[MAX_LINE_LENGTH];
|
||||
*num_sizes = 0;
|
||||
|
||||
while (fgets(line, sizeof(line), file)) {
|
||||
/* Look for lines starting with "Alloc:" */
|
||||
if (strncmp(line, "Alloc:", 6) == 0) {
|
||||
int size;
|
||||
/* Try to extract the size from the line */
|
||||
/* Format: Alloc: 0x55fde046b490 -> 4 (11) at wolfTLSv1_3_client_method_ex:src/tls.c:16714 */
|
||||
if (sscanf(line, "Alloc: %*s -> %d", &size) == 1) {
|
||||
/* Check if this size already exists in our array */
|
||||
int i;
|
||||
for (i = 0; i < *num_sizes; i++) {
|
||||
if (alloc_sizes[i].size == size) {
|
||||
alloc_sizes[i].count++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
/* If not found, add it */
|
||||
if (i == *num_sizes) {
|
||||
if (*num_sizes < MAX_ALLOC_SIZES) {
|
||||
alloc_sizes[*num_sizes].size = size;
|
||||
alloc_sizes[*num_sizes].count = 1;
|
||||
(*num_sizes)++;
|
||||
} else {
|
||||
printf("Warning: Maximum number of allocation sizes reached\n");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Add new size if not found */
|
||||
if (i == *num_allocs && *num_allocs < MAX_ALLOC_SIZES) {
|
||||
allocs[*num_allocs].size = size;
|
||||
allocs[*num_allocs].frequency = 1;
|
||||
(*num_allocs)++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fclose(fp);
|
||||
|
||||
/* Sort allocation sizes from smallest to largest */
|
||||
qsort(allocs, *num_allocs, sizeof(AllocInfo), compare_alloc_sizes);
|
||||
|
||||
fclose(file);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Calculate optimal bucket sizes based on allocation patterns */
|
||||
static int optimize_buckets(AllocInfo* allocs, int num_allocs,
|
||||
BucketInfo* buckets, int* num_buckets) {
|
||||
int i, j;
|
||||
size_t total_waste = 0;
|
||||
*num_buckets = 0;
|
||||
/* Function to compare allocation sizes for sorting */
|
||||
int compare_alloc_sizes(const void* a, const void* b) {
|
||||
return ((AllocSize*)a)->size - ((AllocSize*)b)->size;
|
||||
}
|
||||
|
||||
/* Initialize first bucket with smallest allocation size */
|
||||
if (num_allocs > 0) {
|
||||
buckets[0].size = allocs[0].size > MIN_BUCKET_SIZE ?
|
||||
allocs[0].size : MIN_BUCKET_SIZE;
|
||||
buckets[0].count = allocs[0].frequency;
|
||||
buckets[0].waste = 0;
|
||||
buckets[0].dist = 2; /* Start with small distribution */
|
||||
*num_buckets = 1;
|
||||
/* Function to compare allocation counts for sorting (descending) */
|
||||
int compare_alloc_counts(const void* a, const void* b) {
|
||||
return ((AllocSize*)b)->count - ((AllocSize*)a)->count;
|
||||
}
|
||||
|
||||
/* Function to optimize bucket sizes */
|
||||
void optimize_buckets(AllocSize* alloc_sizes, int num_sizes, int* buckets, int* dist, int* num_buckets) {
|
||||
/* Make a copy of the allocation sizes for frequency sorting */
|
||||
AllocSize* alloc_sizes_by_freq = (AllocSize*)malloc(num_sizes * sizeof(AllocSize));
|
||||
if (!alloc_sizes_by_freq) {
|
||||
printf("Error: Memory allocation failed\n");
|
||||
*num_buckets = 0;
|
||||
return;
|
||||
}
|
||||
|
||||
/* Process remaining allocation sizes */
|
||||
for (i = 1; i < num_allocs; i++) {
|
||||
size_t best_waste = (size_t)-1;
|
||||
int best_bucket = -1;
|
||||
|
||||
/* Find best existing bucket */
|
||||
for (j = 0; j < *num_buckets; j++) {
|
||||
if (allocs[i].size <= buckets[j].size) {
|
||||
size_t waste = (buckets[j].size - allocs[i].size) *
|
||||
allocs[i].frequency;
|
||||
if (waste < best_waste) {
|
||||
best_waste = waste;
|
||||
best_bucket = j;
|
||||
}
|
||||
|
||||
memcpy(alloc_sizes_by_freq, alloc_sizes, num_sizes * sizeof(AllocSize));
|
||||
|
||||
/* Sort by frequency (descending) */
|
||||
qsort(alloc_sizes_by_freq, num_sizes, sizeof(AllocSize), compare_alloc_counts);
|
||||
|
||||
/* Select the most frequent allocation sizes as buckets (up to MAX_BUCKETS) */
|
||||
*num_buckets = (num_sizes < MAX_BUCKETS) ? num_sizes : MAX_BUCKETS;
|
||||
|
||||
/* Copy the selected bucket sizes */
|
||||
for (int i = 0; i < *num_buckets; i++) {
|
||||
buckets[i] = alloc_sizes_by_freq[i].size;
|
||||
/* Distribution is based on frequency */
|
||||
dist[i] = (alloc_sizes_by_freq[i].count > 10) ? 8 :
|
||||
(alloc_sizes_by_freq[i].count > 5) ? 4 : 2;
|
||||
}
|
||||
|
||||
/* Sort buckets by size (ascending) */
|
||||
for (int i = 0; i < *num_buckets - 1; i++) {
|
||||
for (int j = 0; j < *num_buckets - i - 1; j++) {
|
||||
if (buckets[j] > buckets[j + 1]) {
|
||||
/* Swap bucket sizes */
|
||||
int temp = buckets[j];
|
||||
buckets[j] = buckets[j + 1];
|
||||
buckets[j + 1] = temp;
|
||||
|
||||
/* Swap distribution values */
|
||||
temp = dist[j];
|
||||
dist[j] = dist[j + 1];
|
||||
dist[j + 1] = temp;
|
||||
}
|
||||
}
|
||||
|
||||
/* Create new bucket if waste is too high */
|
||||
if (best_bucket == -1 ||
|
||||
(float)best_waste / (allocs[i].size * allocs[i].frequency) >
|
||||
MAX_WASTE_RATIO) {
|
||||
if (*num_buckets < MAX_BUCKETS) {
|
||||
buckets[*num_buckets].size = allocs[i].size;
|
||||
buckets[*num_buckets].count = allocs[i].frequency;
|
||||
buckets[*num_buckets].waste = 0;
|
||||
buckets[*num_buckets].dist = 7; /* Default distribution */
|
||||
(*num_buckets)++;
|
||||
}
|
||||
}
|
||||
else {
|
||||
buckets[best_bucket].count += allocs[i].frequency;
|
||||
buckets[best_bucket].waste += best_waste;
|
||||
total_waste += best_waste;
|
||||
}
|
||||
}
|
||||
|
||||
/* Sort buckets by size */
|
||||
qsort(buckets, *num_buckets, sizeof(BucketInfo), compare_bucket_sizes);
|
||||
|
||||
/* Adjust distribution values based on usage patterns */
|
||||
for (i = 0; i < *num_buckets; i++) {
|
||||
if (buckets[i].count > 100) {
|
||||
buckets[i].dist = 8; /* High usage */
|
||||
}
|
||||
else if (buckets[i].count < 10) {
|
||||
buckets[i].dist = 2; /* Low usage */
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
|
||||
free(alloc_sizes_by_freq);
|
||||
}
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
AllocInfo allocs[MAX_ALLOC_SIZES];
|
||||
BucketInfo buckets[MAX_BUCKETS];
|
||||
int num_allocs = 0;
|
||||
int num_buckets = 0;
|
||||
int i;
|
||||
|
||||
if (!STATIC_MEMORY_ENABLED) {
|
||||
printf("Warning: This tool is designed to work with wolfSSL's static "
|
||||
"memory feature.\n");
|
||||
printf("Please rebuild wolfSSL with --enable-staticmemory to use the "
|
||||
"optimized configurations.\n\n");
|
||||
}
|
||||
|
||||
if (argc != 2) {
|
||||
if (argc < 2) {
|
||||
printf("Usage: %s <memory_log_file>\n", argv[0]);
|
||||
printf("\nExample:\n");
|
||||
printf(" 1. Build wolfSSL with static memory and logging:\n");
|
||||
printf(" cd ../../../wolfssl\n");
|
||||
printf(" ./configure --enable-staticmemory --enable-memorylog\n");
|
||||
printf(" make\n\n");
|
||||
printf(" 2. Run the example client to generate memory logs:\n");
|
||||
printf(" ./examples/client/client -h google.com -d -p 443 -g\n\n");
|
||||
printf(" 3. Run this tool with the memory log file:\n");
|
||||
printf(" %s memory_log.txt\n", argv[0]);
|
||||
return 1;
|
||||
}
|
||||
|
||||
|
||||
printf("Warning: This tool is designed to work with wolfSSL's static memory feature.\n");
|
||||
printf("Please rebuild wolfSSL with --enable-staticmemory to use the optimized configurations.\n\n");
|
||||
|
||||
AllocSize alloc_sizes[MAX_ALLOC_SIZES];
|
||||
int num_sizes = 0;
|
||||
|
||||
/* Parse memory allocation logs */
|
||||
if (parse_memory_logs(argv[1], allocs, &num_allocs) != 0) {
|
||||
if (parse_memory_logs(argv[1], alloc_sizes, &num_sizes) != 0) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
printf("Found %d unique allocation sizes\n\n", num_allocs);
|
||||
|
||||
|
||||
printf("Found %d unique allocation sizes\n\n", num_sizes);
|
||||
|
||||
/* Sort allocation sizes */
|
||||
qsort(alloc_sizes, num_sizes, sizeof(AllocSize), compare_alloc_sizes);
|
||||
|
||||
/* Print allocation sizes and frequencies */
|
||||
printf("Allocation Sizes and Frequencies:\n");
|
||||
printf("Size Count\n");
|
||||
printf("---- -----\n");
|
||||
for (i = 0; i < num_allocs; i++) {
|
||||
printf("%-8zu %d\n", allocs[i].size, allocs[i].frequency);
|
||||
for (int i = 0; i < num_sizes; i++) {
|
||||
printf("%-7d %d\n", alloc_sizes[i].size, alloc_sizes[i].count);
|
||||
}
|
||||
printf("\n");
|
||||
|
||||
|
||||
/* Optimize bucket sizes */
|
||||
if (optimize_buckets(allocs, num_allocs, buckets, &num_buckets) != 0) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Print optimized bucket configuration */
|
||||
int buckets[MAX_BUCKETS];
|
||||
int dist[MAX_BUCKETS];
|
||||
int num_buckets = 0;
|
||||
|
||||
optimize_buckets(alloc_sizes, num_sizes, buckets, dist, &num_buckets);
|
||||
|
||||
/* Print optimized bucket sizes and distribution */
|
||||
printf("Optimized Bucket Sizes and Distribution:\n");
|
||||
printf("Size Count Waste Dist\n");
|
||||
printf("---- ----- ----- ----\n");
|
||||
for (i = 0; i < num_buckets; i++) {
|
||||
printf("%-8zu %-7d %-7zu %d\n",
|
||||
buckets[i].size, buckets[i].count,
|
||||
buckets[i].waste, buckets[i].dist);
|
||||
printf("Size Count Wasted Dist\n");
|
||||
printf("---- ----- ------ ----\n");
|
||||
|
||||
float total_waste = 0.0;
|
||||
for (int i = 0; i < num_buckets; i++) {
|
||||
/* Find the closest allocation size that fits in this bucket */
|
||||
int closest_size = 0;
|
||||
int closest_count = 0;
|
||||
for (int j = 0; j < num_sizes; j++) {
|
||||
if (alloc_sizes[j].size <= buckets[i] &&
|
||||
alloc_sizes[j].size > closest_size) {
|
||||
closest_size = alloc_sizes[j].size;
|
||||
closest_count = alloc_sizes[j].count;
|
||||
}
|
||||
}
|
||||
|
||||
float waste = (float)(buckets[i] - closest_size);
|
||||
total_waste += waste * closest_count;
|
||||
|
||||
printf("%-7d %-7d %-7.2f %d\n", buckets[i], closest_count,
|
||||
waste, dist[i]);
|
||||
}
|
||||
|
||||
printf("\nTotal waste: %.2f bytes\n\n", total_waste);
|
||||
|
||||
/* Print WOLFMEM_BUCKETS and WOLFMEM_DIST macros */
|
||||
printf("WOLFMEM_BUCKETS and WOLFMEM_DIST Macros:\n");
|
||||
printf("#define WOLFMEM_BUCKETS ");
|
||||
for (int i = 0; i < num_buckets; i++) {
|
||||
printf("%d", buckets[i]);
|
||||
if (i < num_buckets - 1) {
|
||||
printf(",");
|
||||
}
|
||||
}
|
||||
printf("\n");
|
||||
|
||||
/* Print WOLFMEM_BUCKETS and WOLFMEM_DIST macros */
|
||||
printf("#define WOLFMEM_BUCKETS ");
|
||||
for (i = 0; i < num_buckets; i++) {
|
||||
printf("%zu%s", buckets[i].size,
|
||||
i < num_buckets - 1 ? "," : "\n");
|
||||
}
|
||||
|
||||
|
||||
printf("#define WOLFMEM_DIST ");
|
||||
for (i = 0; i < num_buckets; i++) {
|
||||
printf("%d%s", buckets[i].dist,
|
||||
i < num_buckets - 1 ? "," : "\n");
|
||||
for (int i = 0; i < num_buckets; i++) {
|
||||
printf("%d", dist[i]);
|
||||
if (i < num_buckets - 1) {
|
||||
printf(",");
|
||||
}
|
||||
}
|
||||
|
||||
printf("\n");
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,135 +1,44 @@
|
|||
#!/bin/bash
|
||||
# test_operations.sh
|
||||
#
|
||||
# Copyright (C) 2006-2025 wolfSSL Inc.
|
||||
#
|
||||
# This file is part of wolfSSL.
|
||||
#
|
||||
# wolfSSL is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# wolfSSL is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335, USA
|
||||
|
||||
# Default values
|
||||
WOLFSSL_DIR="../../../wolfssl"
|
||||
RESULTS_DIR="test_results"
|
||||
# Script to test the memory bucket optimizer with different TLS operations
|
||||
|
||||
# Test configurations
|
||||
declare -A TESTS=(
|
||||
["tls12_google"]="google.com:443"
|
||||
["tls13_google"]="google.com:443:-v 4"
|
||||
["tls12_cloudflare"]="cloudflare.com:443"
|
||||
["tls13_cloudflare"]="cloudflare.com:443:-v 4"
|
||||
["tls12_resume"]="google.com:443:-r"
|
||||
["tls13_resume"]="google.com:443:-v 4 -r"
|
||||
["tls12_nonblocking"]="google.com:443:-N"
|
||||
["tls13_nonblocking"]="google.com:443:-v 4 -N"
|
||||
)
|
||||
# Set up directories
|
||||
WOLFSSL_DIR=~/repos/wolfssl
|
||||
RESULTS_DIR=./results
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
|
||||
# Create results directory
|
||||
mkdir -p "$RESULTS_DIR"
|
||||
mkdir -p $RESULTS_DIR
|
||||
|
||||
# Check if wolfSSL is built with memory logging
|
||||
if [ ! -f "$WOLFSSL_DIR/options.h" ]; then
|
||||
echo "Error: wolfSSL options.h not found at $WOLFSSL_DIR/options.h"
|
||||
echo "Please make sure wolfSSL is built with --enable-memorylog"
|
||||
exit 1
|
||||
fi
|
||||
# Build wolfSSL with memory logging enabled
|
||||
cd $WOLFSSL_DIR || exit 1
|
||||
./autogen.sh && ./configure --enable-memorylog --enable-staticmemory && make
|
||||
|
||||
# Check if memory logging is enabled
|
||||
if ! grep -q "WOLFSSL_MEMORY_LOG" "$WOLFSSL_DIR/options.h"; then
|
||||
echo "Warning: wolfSSL is not built with memory logging enabled"
|
||||
echo "Using sample memory logs for testing"
|
||||
# Function to run a test and collect memory usage data
|
||||
run_test() {
|
||||
local test_name="$1"
|
||||
local host="$2"
|
||||
local port="$3"
|
||||
local extra_args="$4"
|
||||
|
||||
# Create sample memory logs
|
||||
mkdir -p "$RESULTS_DIR/samples"
|
||||
echo "Running test: $test_name"
|
||||
echo "Host: $host, Port: $port, Extra args: $extra_args"
|
||||
|
||||
# Sample TLS 1.2 memory log
|
||||
cat > "$RESULTS_DIR/samples/tls12_sample.txt" << 'SAMPLE'
|
||||
Alloc: 16 bytes at address: 0x55b7a2c31e70 from func: wolfSSL_Init
|
||||
Alloc: 24 bytes at address: 0x55b7a2c31e90 from func: wolfSSL_CTX_new
|
||||
Alloc: 32 bytes at address: 0x55b7a2c31eb0 from func: wolfSSL_new
|
||||
Alloc: 64 bytes at address: 0x55b7a2c31ef0 from func: wolfSSL_connect
|
||||
Alloc: 128 bytes at address: 0x55b7a2c31f40 from func: DoHandShake
|
||||
Alloc: 256 bytes at address: 0x55b7a2c31fc0 from func: BuildMessage
|
||||
Alloc: 512 bytes at address: 0x55b7a2c320d0 from func: BuildCertHashes
|
||||
Alloc: 1024 bytes at address: 0x55b7a2c322f0 from func: BuildFinished
|
||||
Alloc: 2048 bytes at address: 0x55b7a2c32730 from func: BuildTls13Message
|
||||
Alloc: 4096 bytes at address: 0x55b7a2c32fb0 from func: BuildTls13Message
|
||||
SAMPLE
|
||||
# Run the example client with memory logging
|
||||
$WOLFSSL_DIR/examples/client/client -h "$host" -d -p "$port" -g $extra_args > "$RESULTS_DIR/${test_name}_output.txt" 2>&1
|
||||
|
||||
# Sample TLS 1.3 memory log
|
||||
cat > "$RESULTS_DIR/samples/tls13_sample.txt" << 'SAMPLE'
|
||||
Alloc: 16 bytes at address: 0x55b7a2c31e70 from func: wolfSSL_Init
|
||||
Alloc: 24 bytes at address: 0x55b7a2c31e90 from func: wolfSSL_CTX_new
|
||||
Alloc: 32 bytes at address: 0x55b7a2c31eb0 from func: wolfSSL_new
|
||||
Alloc: 64 bytes at address: 0x55b7a2c31ef0 from func: wolfSSL_connect
|
||||
Alloc: 128 bytes at address: 0x55b7a2c31f40 from func: DoHandShake
|
||||
Alloc: 256 bytes at address: 0x55b7a2c31fc0 from func: BuildMessage
|
||||
Alloc: 512 bytes at address: 0x55b7a2c320d0 from func: BuildCertHashes
|
||||
Alloc: 1024 bytes at address: 0x55b7a2c322f0 from func: BuildFinished
|
||||
Alloc: 2048 bytes at address: 0x55b7a2c32730 from func: BuildTls13Message
|
||||
Alloc: 4096 bytes at address: 0x55b7a2c32fb0 from func: BuildTls13Message
|
||||
Alloc: 8192 bytes at address: 0x55b7a2c34fb0 from func: BuildTls13KeyShare
|
||||
SAMPLE
|
||||
|
||||
# Use sample logs for testing
|
||||
for test_name in "${!TESTS[@]}"; do
|
||||
if [[ "$test_name" == *"tls13"* ]]; then
|
||||
cp "$RESULTS_DIR/samples/tls13_sample.txt" "$RESULTS_DIR/${test_name}_memory.txt"
|
||||
else
|
||||
cp "$RESULTS_DIR/samples/tls12_sample.txt" "$RESULTS_DIR/${test_name}_memory.txt"
|
||||
fi
|
||||
done
|
||||
else
|
||||
# Build wolfSSL with memory logging enabled
|
||||
echo "Building wolfSSL with memory logging..."
|
||||
cd "$WOLFSSL_DIR" || exit 1
|
||||
./autogen.sh
|
||||
./configure --enable-memorylog
|
||||
make
|
||||
|
||||
# Run tests for each configuration
|
||||
for test_name in "${!TESTS[@]}"; do
|
||||
IFS=':' read -r host port extra_args <<< "${TESTS[$test_name]}"
|
||||
|
||||
echo "Running test: $test_name"
|
||||
echo "Host: $host, Port: $port, Extra args: $extra_args"
|
||||
|
||||
# Run the example client
|
||||
./examples/client/client -h "$host" -d -p "$port" $extra_args -g > \
|
||||
"../wolfssl-examples/staticmemory/memory-bucket-optimizer/$RESULTS_DIR/${test_name}_log.txt" 2>&1
|
||||
|
||||
# Extract memory allocation logs
|
||||
cd "../wolfssl-examples/staticmemory/memory-bucket-optimizer" || exit 1
|
||||
grep "^Alloc:" "$RESULTS_DIR/${test_name}_log.txt" > "$RESULTS_DIR/${test_name}_memory.txt"
|
||||
|
||||
# Return to wolfSSL directory for next test
|
||||
cd "$WOLFSSL_DIR" || exit 1
|
||||
done
|
||||
fi
|
||||
|
||||
# Run the memory bucket optimizer for each test
|
||||
cd "../wolfssl-examples/staticmemory/memory-bucket-optimizer" || exit 1
|
||||
for test_name in "${!TESTS[@]}"; do
|
||||
echo "Optimizing memory buckets for $test_name..."
|
||||
# Extract memory allocation logs
|
||||
grep "^Alloc:" "$RESULTS_DIR/${test_name}_output.txt" > "$RESULTS_DIR/${test_name}_memory.txt"
|
||||
|
||||
# Run the memory bucket optimizer
|
||||
cd src || exit 1
|
||||
./memory_bucket_optimizer "../../$RESULTS_DIR/${test_name}_memory.txt" > \
|
||||
"../../$RESULTS_DIR/${test_name}_buckets.txt"
|
||||
|
||||
cd .. || exit 1
|
||||
done
|
||||
$SCRIPT_DIR/src/memory_bucket_optimizer "$RESULTS_DIR/${test_name}_memory.txt" > "$RESULTS_DIR/${test_name}_buckets.txt"
|
||||
}
|
||||
|
||||
# Run tests for different TLS operations
|
||||
run_test "tls12_google" "google.com" "443" ""
|
||||
run_test "tls13_google" "google.com" "443" "-v 4"
|
||||
run_test "tls12_cloudflare" "cloudflare.com" "443" ""
|
||||
run_test "tls13_cloudflare" "cloudflare.com" "443" "-v 4"
|
||||
|
||||
# Generate a summary of the results
|
||||
echo "Generating summary..."
|
||||
|
|
@ -147,8 +56,20 @@ for result_file in "$RESULTS_DIR"/*_buckets.txt; do
|
|||
echo "$test_name,$total_allocs,$unique_sizes,$largest_bucket,$total_waste" >> "$RESULTS_DIR/summary.csv"
|
||||
done
|
||||
|
||||
# Print the summary
|
||||
echo "Test results summary:"
|
||||
cat "$RESULTS_DIR/summary.csv"
|
||||
|
||||
echo "All tests completed. Results saved in $RESULTS_DIR/"
|
||||
echo "Summary saved to $RESULTS_DIR/summary.csv"
|
||||
|
||||
# Compare the different bucket configurations
|
||||
echo "Comparing bucket configurations for different TLS operations..."
|
||||
echo
|
||||
|
||||
for result_file in "$RESULTS_DIR"/*_buckets.txt; do
|
||||
test_name=$(basename "$result_file" _buckets.txt)
|
||||
buckets=$(grep "^#define WOLFMEM_BUCKETS" "$result_file" | sed 's/#define WOLFMEM_BUCKETS //' || echo "")
|
||||
dist=$(grep "^#define WOLFMEM_DIST" "$result_file" | sed 's/#define WOLFMEM_DIST //' || echo "")
|
||||
|
||||
echo "$test_name:"
|
||||
echo " WOLFMEM_BUCKETS: $buckets"
|
||||
echo " WOLFMEM_DIST: $dist"
|
||||
echo
|
||||
done
|
||||
|
|
|
|||
|
|
@ -0,0 +1,79 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Script to test the memory bucket optimizer with different TLS operations
|
||||
|
||||
# Set up directories
|
||||
WOLFSSL_DIR=~/repos/wolfssl
|
||||
RESULTS_DIR=./results
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
|
||||
# Create results directory
|
||||
mkdir -p $RESULTS_DIR
|
||||
|
||||
# Build wolfSSL with memory logging enabled
|
||||
cd $WOLFSSL_DIR || exit 1
|
||||
./autogen.sh && ./configure --enable-memorylog --enable-staticmemory && make
|
||||
|
||||
# Function to run a test and collect memory usage data
|
||||
run_test() {
|
||||
local test_name="$1"
|
||||
local host="$2"
|
||||
local port="$3"
|
||||
local extra_args="$4"
|
||||
|
||||
echo "Running test: $test_name"
|
||||
echo "Host: $host, Port: $port, Extra args: $extra_args"
|
||||
|
||||
# Run the example client with memory logging
|
||||
$WOLFSSL_DIR/examples/client/client -h "$host" -d -p "$port" -g $extra_args > "$RESULTS_DIR/${test_name}_output.txt" 2>&1
|
||||
|
||||
# Extract memory allocation logs
|
||||
grep "^Alloc:" "$RESULTS_DIR/${test_name}_output.txt" > "$RESULTS_DIR/${test_name}_memory.txt"
|
||||
|
||||
# Run the memory bucket optimizer
|
||||
cd $SCRIPT_DIR/src || exit 1
|
||||
./memory_bucket_optimizer "$RESULTS_DIR/${test_name}_memory.txt" > "$SCRIPT_DIR/$RESULTS_DIR/${test_name}_buckets.txt"
|
||||
|
||||
# Return to script directory
|
||||
cd $SCRIPT_DIR || exit 1
|
||||
}
|
||||
|
||||
# Run tests for different TLS operations
|
||||
run_test "tls12_google" "google.com" "443" ""
|
||||
run_test "tls13_google" "google.com" "443" "-v 4"
|
||||
run_test "tls12_cloudflare" "cloudflare.com" "443" ""
|
||||
run_test "tls13_cloudflare" "cloudflare.com" "443" "-v 4"
|
||||
|
||||
# Generate a summary of the results
|
||||
echo "Generating summary..."
|
||||
echo "Test Name,Total Allocs,Unique Sizes,Largest Bucket,Total Waste" > "$RESULTS_DIR/summary.csv"
|
||||
|
||||
for result_file in "$RESULTS_DIR"/*_buckets.txt; do
|
||||
test_name=$(basename "$result_file" _buckets.txt)
|
||||
alloc_file="$RESULTS_DIR/${test_name}_memory.txt"
|
||||
|
||||
total_allocs=$(grep -c "^Alloc:" "$alloc_file" || echo "0")
|
||||
unique_sizes=$(grep "Found .* unique allocation sizes" "$result_file" | awk '{print $2}' || echo "0")
|
||||
largest_bucket=$(grep -A 3 "Optimized Bucket Sizes" "$result_file" | tail -n 1 | awk '{print $1}' || echo "0")
|
||||
total_waste=$(grep -A 3 "Optimized Bucket Sizes" "$result_file" | tail -n 1 | awk '{print $3}' || echo "0")
|
||||
|
||||
echo "$test_name,$total_allocs,$unique_sizes,$largest_bucket,$total_waste" >> "$RESULTS_DIR/summary.csv"
|
||||
done
|
||||
|
||||
echo "All tests completed. Results saved in $RESULTS_DIR/"
|
||||
echo "Summary saved to $RESULTS_DIR/summary.csv"
|
||||
|
||||
# Compare the different bucket configurations
|
||||
echo "Comparing bucket configurations for different TLS operations..."
|
||||
echo
|
||||
|
||||
for result_file in "$RESULTS_DIR"/*_buckets.txt; do
|
||||
test_name=$(basename "$result_file" _buckets.txt)
|
||||
buckets=$(grep "^#define WOLFMEM_BUCKETS" "$result_file" | sed 's/#define WOLFMEM_BUCKETS //' || echo "")
|
||||
dist=$(grep "^#define WOLFMEM_DIST" "$result_file" | sed 's/#define WOLFMEM_DIST //' || echo "")
|
||||
|
||||
echo "$test_name:"
|
||||
echo " WOLFMEM_BUCKETS: $buckets"
|
||||
echo " WOLFMEM_DIST: $dist"
|
||||
echo
|
||||
done
|
||||
|
|
@ -1,46 +1,36 @@
|
|||
# Memory Bucket Optimizer Visualization
|
||||
|
||||
This directory contains scripts for visualizing memory allocation patterns and bucket optimization results.
|
||||
This directory contains scripts and data files for visualizing memory bucket optimization results.
|
||||
|
||||
## Visualization Scripts
|
||||
|
||||
1. `allocation_histogram.gp`: Generates a histogram of allocation sizes and frequencies
|
||||
2. `bucket_optimization.gp`: Visualizes bucket sizes and waste
|
||||
3. `tls_comparison.gp`: Compares memory usage across different TLS operations
|
||||
4. `memory_usage_over_time.gp`: Tracks memory usage over time
|
||||
5. `memory_heatmap.gp`: Visualizes memory usage patterns
|
||||
- `generate_data.sh`: Main script to generate data files and plots
|
||||
- `allocation_histogram.gp`: Gnuplot script for allocation size histogram
|
||||
- `bucket_optimization.gp`: Gnuplot script for bucket optimization visualization
|
||||
- `tls_comparison.gp`: Gnuplot script for TLS operation comparison
|
||||
|
||||
## Usage
|
||||
## Generated Plots
|
||||
|
||||
```bash
|
||||
# Generate all visualization plots
|
||||
./generate_data.sh
|
||||
|
||||
# Generate a specific plot
|
||||
gnuplot allocation_histogram.gp
|
||||
```
|
||||
|
||||
## Output Files
|
||||
|
||||
The scripts generate PNG image files:
|
||||
|
||||
- `allocation_histogram.png`: Histogram of allocation sizes
|
||||
- `bucket_optimization.png`: Bucket sizes and waste
|
||||
- `tls_comparison.png`: Comparison of TLS operations
|
||||
- `memory_usage_over_time.png`: Memory usage over time
|
||||
- `memory_heatmap.png`: Memory usage heatmap
|
||||
- `tls12_google_allocation_histogram.png`: Allocation size distribution for TLS 1.2 connection to Google
|
||||
- `tls13_google_allocation_histogram.png`: Allocation size distribution for TLS 1.3 connection to Google
|
||||
- `tls12_cloudflare_allocation_histogram.png`: Allocation size distribution for TLS 1.2 connection to Cloudflare
|
||||
- `tls13_cloudflare_allocation_histogram.png`: Allocation size distribution for TLS 1.3 connection to Cloudflare
|
||||
- `tls12_google_bucket_optimization.png`: Bucket optimization for TLS 1.2 connection to Google
|
||||
- `tls13_google_bucket_optimization.png`: Bucket optimization for TLS 1.3 connection to Google
|
||||
- `tls12_cloudflare_bucket_optimization.png`: Bucket optimization for TLS 1.2 connection to Cloudflare
|
||||
- `tls13_cloudflare_bucket_optimization.png`: Bucket optimization for TLS 1.3 connection to Cloudflare
|
||||
- `tls_comparison.png`: Comparison of memory usage for different TLS operations
|
||||
|
||||
## Data Files
|
||||
|
||||
The scripts use data files generated from the memory bucket optimizer results:
|
||||
The `data` directory contains intermediate data files used for generating the plots.
|
||||
|
||||
- `allocation_sizes.txt`: Allocation sizes and frequencies
|
||||
- `bucket_sizes.txt`: Bucket sizes and waste
|
||||
- `tls_comparison.txt`: Memory usage for different TLS operations
|
||||
- `memory_usage.txt`: Memory usage over time
|
||||
- `memory_heatmap.txt`: Memory usage patterns
|
||||
## Usage
|
||||
|
||||
## Requirements
|
||||
To generate all plots, run:
|
||||
|
||||
- gnuplot 5.2 or later
|
||||
- bash 5.0 or later
|
||||
```bash
|
||||
./generate_data.sh
|
||||
```
|
||||
|
||||
This will process the memory logs and bucket configurations in the `results` directory and generate the plots in this directory.
|
||||
|
|
|
|||
|
|
@ -1,18 +1,10 @@
|
|||
# Gnuplot script to visualize allocation sizes and frequencies
|
||||
|
||||
# Set the output file format
|
||||
set terminal png size 800,600
|
||||
set output "allocation_histogram.png"
|
||||
|
||||
# Set the title and labels
|
||||
set title "Memory Allocation Sizes and Frequencies"
|
||||
set output "/home/ubuntu/repos/wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization/tls13_cloudflare_allocation_histogram.png"
|
||||
set title "Allocation Size Distribution for tls13_cloudflare"
|
||||
set xlabel "Allocation Size (bytes)"
|
||||
set ylabel "Frequency"
|
||||
set grid
|
||||
|
||||
# Set the style
|
||||
set style fill solid 0.5
|
||||
set boxwidth 0.8
|
||||
|
||||
# Plot the data
|
||||
plot "allocation_data.txt" using 1:2 with boxes title "Allocation Frequency"
|
||||
set grid
|
||||
set logscale y
|
||||
plot "/home/ubuntu/repos/wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization/data/tls13_cloudflare_alloc_data.txt" using 2:1 with boxes title "Allocation Sizes"
|
||||
|
|
|
|||
|
|
@ -1,24 +1,11 @@
|
|||
# Gnuplot script to visualize bucket sizes and waste
|
||||
|
||||
# Set the output file format
|
||||
set terminal png size 800,600
|
||||
set output "bucket_optimization.png"
|
||||
|
||||
# Set the title and labels
|
||||
set title "Memory Bucket Sizes and Waste"
|
||||
set xlabel "Bucket Size (bytes)"
|
||||
set ylabel "Waste (bytes)"
|
||||
set y2label "Distribution"
|
||||
set output "/home/ubuntu/repos/wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization/tls13_cloudflare_bucket_optimization.png"
|
||||
set title "Bucket Optimization for tls13_cloudflare"
|
||||
set xlabel "Size (bytes)"
|
||||
set ylabel "Count"
|
||||
set grid
|
||||
|
||||
# Set the style
|
||||
set style fill solid 0.5
|
||||
set boxwidth 0.8
|
||||
|
||||
# Enable y2 axis
|
||||
set ytics nomirror
|
||||
set y2tics
|
||||
|
||||
# Plot the data
|
||||
plot "bucket_data.txt" using 1:3 with boxes title "Waste" axes x1y1, \
|
||||
"bucket_data.txt" using 1:4 with linespoints title "Distribution" axes x1y2
|
||||
set key outside
|
||||
plot "/home/ubuntu/repos/wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization/data/tls13_cloudflare_alloc_data.txt" using 2:1 with boxes title "Allocation Sizes", \
|
||||
"/home/ubuntu/repos/wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization/data/bucket_sizes.txt" using 1:(0.5) with impulses lw 2 title "Bucket Sizes"
|
||||
|
|
|
|||
|
|
@ -1,58 +1,125 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Script to generate data files for gnuplot visualization
|
||||
# Script to generate data files and plots for memory bucket optimization
|
||||
|
||||
# Check if results directory exists
|
||||
if [ ! -d "../verification_results" ]; then
|
||||
echo "Error: verification_results directory not found"
|
||||
exit 1
|
||||
fi
|
||||
# Set up directories
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
RESULTS_DIR="$SCRIPT_DIR/../results"
|
||||
|
||||
# Create allocation data file
|
||||
echo "# Allocation Size Frequency" > allocation_data.txt
|
||||
grep -A 100 "Allocation Sizes and Frequencies:" ../verification_results/tls12_google_buckets.txt | grep -A 100 "Size" | grep -v "Size" | grep -v "----" | grep -v "^$" | grep -v "Optimized" | awk '{print $1, $2}' > allocation_data.txt
|
||||
# Create data directory
|
||||
mkdir -p "$SCRIPT_DIR/data"
|
||||
|
||||
# Create bucket data file
|
||||
echo "# Bucket Size Count Waste Dist" > bucket_data.txt
|
||||
grep -A 100 "Optimized Bucket Sizes and Distribution:" ../verification_results/tls12_google_buckets.txt | grep -A 100 "Size" | grep -v "Size" | grep -v "----" | grep -v "^$" | grep -v "WOLFMEM" | awk '{print $1, $2, $3, $4}' > bucket_data.txt
|
||||
# Function to extract allocation sizes and frequencies from memory logs
|
||||
extract_allocation_data() {
|
||||
local memory_log="$1"
|
||||
local output_file="$2"
|
||||
|
||||
# Extract allocation sizes and count occurrences
|
||||
grep "^Alloc:" "$memory_log" | sed -n 's/.*-> \([0-9]\+\).*/\1/p' | sort -n | uniq -c > "$output_file"
|
||||
}
|
||||
|
||||
# Create TLS comparison data file
|
||||
echo "# TLS Operation Total_Allocs Unique_Sizes Largest_Bucket Total_Waste" > tls_comparison.txt
|
||||
cat ../verification_results/summary.csv | grep -v "Test Name" | sed 's/,/ /g' > tls_comparison.txt
|
||||
# Function to generate gnuplot script for allocation histogram
|
||||
generate_allocation_histogram() {
|
||||
local data_file="$1"
|
||||
local output_file="$2"
|
||||
local title="$3"
|
||||
|
||||
cat > "$SCRIPT_DIR/allocation_histogram.gp" << EOL
|
||||
set terminal png size 800,600
|
||||
set output "$output_file"
|
||||
set title "$title"
|
||||
set xlabel "Allocation Size (bytes)"
|
||||
set ylabel "Frequency"
|
||||
set style fill solid 0.5
|
||||
set boxwidth 0.8
|
||||
set grid
|
||||
set logscale y
|
||||
plot "$data_file" using 2:1 with boxes title "Allocation Sizes"
|
||||
EOL
|
||||
|
||||
gnuplot "$SCRIPT_DIR/allocation_histogram.gp"
|
||||
}
|
||||
|
||||
# Create sample memory usage over time data file (this would be replaced with actual data)
|
||||
echo "# Time Default_Config Optimized_Config" > memory_usage_over_time.txt
|
||||
echo "0 0 0" >> memory_usage_over_time.txt
|
||||
echo "1 1000 800" >> memory_usage_over_time.txt
|
||||
echo "2 2000 1600" >> memory_usage_over_time.txt
|
||||
echo "3 3000 2400" >> memory_usage_over_time.txt
|
||||
echo "4 4000 3200" >> memory_usage_over_time.txt
|
||||
echo "5 5000 4000" >> memory_usage_over_time.txt
|
||||
# Function to generate gnuplot script for bucket optimization
|
||||
generate_bucket_optimization() {
|
||||
local data_file="$1"
|
||||
local buckets_file="$2"
|
||||
local output_file="$3"
|
||||
local title="$4"
|
||||
|
||||
# Extract bucket sizes from buckets file
|
||||
grep "^#define WOLFMEM_BUCKETS" "$buckets_file" | sed 's/#define WOLFMEM_BUCKETS //' | tr ',' '\n' > "$SCRIPT_DIR/data/bucket_sizes.txt"
|
||||
|
||||
cat > "$SCRIPT_DIR/bucket_optimization.gp" << EOL
|
||||
set terminal png size 800,600
|
||||
set output "$output_file"
|
||||
set title "$title"
|
||||
set xlabel "Size (bytes)"
|
||||
set ylabel "Count"
|
||||
set grid
|
||||
set style fill solid 0.5
|
||||
set boxwidth 0.8
|
||||
set key outside
|
||||
plot "$data_file" using 2:1 with boxes title "Allocation Sizes", \\
|
||||
"$SCRIPT_DIR/data/bucket_sizes.txt" using 1:(0.5) with impulses lw 2 title "Bucket Sizes"
|
||||
EOL
|
||||
|
||||
gnuplot "$SCRIPT_DIR/bucket_optimization.gp"
|
||||
}
|
||||
|
||||
# Create sample memory heatmap data file (this would be replaced with actual data)
|
||||
echo "# BucketSize Operation MemoryUsage" > memory_heatmap.txt
|
||||
echo "16 1 1000" >> memory_heatmap.txt
|
||||
echo "32 1 2000" >> memory_heatmap.txt
|
||||
echo "64 1 3000" >> memory_heatmap.txt
|
||||
echo "128 1 4000" >> memory_heatmap.txt
|
||||
echo "16 2 800" >> memory_heatmap.txt
|
||||
echo "32 2 1600" >> memory_heatmap.txt
|
||||
echo "64 2 2400" >> memory_heatmap.txt
|
||||
echo "128 2 3200" >> memory_heatmap.txt
|
||||
echo "16 3 600" >> memory_heatmap.txt
|
||||
echo "32 3 1200" >> memory_heatmap.txt
|
||||
echo "64 3 1800" >> memory_heatmap.txt
|
||||
echo "128 3 2400" >> memory_heatmap.txt
|
||||
echo "16 4 400" >> memory_heatmap.txt
|
||||
echo "32 4 800" >> memory_heatmap.txt
|
||||
echo "64 4 1200" >> memory_heatmap.txt
|
||||
echo "128 4 1600" >> memory_heatmap.txt
|
||||
# Function to generate gnuplot script for TLS operation comparison
|
||||
generate_tls_comparison() {
|
||||
local output_file="$1"
|
||||
|
||||
# Create data file for TLS operation comparison
|
||||
echo "# Operation TotalAllocs UniqueSizes LargestBucket TotalWaste" > "$SCRIPT_DIR/data/tls_comparison.txt"
|
||||
grep -v "Test Name" "$RESULTS_DIR/summary.csv" | sed 's/,/ /g' >> "$SCRIPT_DIR/data/tls_comparison.txt"
|
||||
|
||||
cat > "$SCRIPT_DIR/tls_comparison.gp" << EOL
|
||||
set terminal png size 800,600
|
||||
set output "$output_file"
|
||||
set title "TLS Operation Comparison"
|
||||
set style data histogram
|
||||
set style histogram cluster gap 1
|
||||
set style fill solid 0.5 border -1
|
||||
set boxwidth 0.9
|
||||
set xtics rotate by -45
|
||||
set grid
|
||||
set key outside
|
||||
plot "$SCRIPT_DIR/data/tls_comparison.txt" using 2:xtic(1) title "Total Allocs", \\
|
||||
"" using 3 title "Unique Sizes", \\
|
||||
"" using 4 title "Largest Bucket", \\
|
||||
"" using 5 title "Total Waste"
|
||||
EOL
|
||||
|
||||
gnuplot "$SCRIPT_DIR/tls_comparison.gp"
|
||||
}
|
||||
|
||||
# Generate the plots
|
||||
gnuplot allocation_histogram.gp
|
||||
gnuplot bucket_optimization.gp
|
||||
gnuplot tls_comparison.gp
|
||||
gnuplot memory_usage_over_time.gp
|
||||
gnuplot memory_heatmap.gp
|
||||
# Process each TLS operation
|
||||
for test_name in tls12_google tls13_google tls12_cloudflare tls13_cloudflare; do
|
||||
memory_log="$RESULTS_DIR/${test_name}_memory.txt"
|
||||
buckets_file="$RESULTS_DIR/${test_name}_buckets.txt"
|
||||
|
||||
if [ -s "$memory_log" ]; then
|
||||
# Extract allocation data
|
||||
extract_allocation_data "$memory_log" "$SCRIPT_DIR/data/${test_name}_alloc_data.txt"
|
||||
|
||||
# Generate allocation histogram
|
||||
generate_allocation_histogram "$SCRIPT_DIR/data/${test_name}_alloc_data.txt" \
|
||||
"$SCRIPT_DIR/${test_name}_allocation_histogram.png" \
|
||||
"Allocation Size Distribution for ${test_name}"
|
||||
|
||||
# Generate bucket optimization plot
|
||||
if [ -s "$buckets_file" ]; then
|
||||
generate_bucket_optimization "$SCRIPT_DIR/data/${test_name}_alloc_data.txt" \
|
||||
"$buckets_file" \
|
||||
"$SCRIPT_DIR/${test_name}_bucket_optimization.png" \
|
||||
"Bucket Optimization for ${test_name}"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Data files and plots generated successfully"
|
||||
# Generate TLS operation comparison
|
||||
generate_tls_comparison "$SCRIPT_DIR/tls_comparison.png"
|
||||
|
||||
echo "Visualization completed. Results saved in $SCRIPT_DIR/"
|
||||
|
|
|
|||
|
|
@ -1,22 +1,14 @@
|
|||
# Gnuplot script to compare different TLS operations
|
||||
|
||||
# Set the output file format
|
||||
set terminal png size 800,600
|
||||
set output "tls_comparison.png"
|
||||
|
||||
# Set the title and labels
|
||||
set title "Memory Usage Comparison for Different TLS Operations"
|
||||
set xlabel "TLS Operation"
|
||||
set ylabel "Memory Usage"
|
||||
set grid
|
||||
|
||||
# Set the style
|
||||
set output "/home/ubuntu/repos/wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization/tls_comparison.png"
|
||||
set title "TLS Operation Comparison"
|
||||
set style data histogram
|
||||
set style histogram cluster gap 1
|
||||
set style fill solid 0.5 border -1
|
||||
set boxwidth 0.9
|
||||
|
||||
# Plot the data
|
||||
plot "tls_comparison.txt" using 2:xtic(1) title "Total Allocations", \
|
||||
set xtics rotate by -45
|
||||
set grid
|
||||
set key outside
|
||||
plot "/home/ubuntu/repos/wolfssl-examples/staticmemory/memory-bucket-optimizer/visualization/data/tls_comparison.txt" using 2:xtic(1) title "Total Allocs", \
|
||||
"" using 3 title "Unique Sizes", \
|
||||
"" using 4 title "Largest Bucket", \
|
||||
"" using 5 title "Total Waste"
|
||||
|
|
|
|||
Loading…
Reference in New Issue