Merge branch 'develop' into kgb-logging

README.md

@@ -1,14 +1,12 @@
-
 # FreeDATA
+
 > FreeDATA is a versatile, **open-source platform designed specifically for HF communications**, leveraging **codec2** data modes for robust global digital communication. It features a network-based server-client architecture, a REST API, multi-platform compatibility, and a messaging system.
 
 > Please keep in mind, this project is still **under development** with many issues which need to be solved.
 
-
 [![CodeFactor](https://www.codefactor.io/repository/github/dj2ls/freedata/badge)](https://www.codefactor.io/repository/github/dj2ls/freedata)
 [![Modem tests](https://github.com/DJ2LS/FreeDATA/actions/workflows/modem_tests.yml/badge.svg)](https://github.com/DJ2LS/FreeDATA/actions/workflows/modem_tests.yml)
 
-
 ![FreeDATA_main_screen.png](documentation%2FFreeDATA_main_screen.png)
 
 ![FreeDATA_chat_screen.png](documentation%2FFreeDATA_chat_screen.png)

freedata_gui/package.json

@@ -1,6 +1,6 @@
 {
   "name": "FreeDATA",
-  "version": "0.16.12-alpha",
+  "version": "0.17.0-beta",
   "description": "FreeDATA Client application for connecting to FreeDATA server",
   "private": true,
   "scripts": {

freedata_gui/src/components/dynamic_components.vue

@@ -754,7 +754,7 @@ onMounted(() => {
       </button>
       <hr/>
       <button
-        class="btn btn-sm btn-outline-dark"
+        class="btn btn-sm btn-outline-secondary"
         type="button"
         @click="loadPreset"
         :title="$t('grid.restorepreset_help')"
@@ -763,7 +763,7 @@ onMounted(() => {
         {{ $t('grid.restorepreset') }}
       </button>&nbsp;
       <button
-        class="btn btn-sm btn-outline-dark"
+        class="btn btn-sm btn-outline-secondary"
         type="button"
         @click="savePreset"
         :title="$t('grid.savepreset_help')"

freedata_gui/src/components/grid/grid_active_stats.vue

@@ -74,7 +74,22 @@ const transmissionSpeedChartOptions = {
         drawOnChartArea: false, // only want the grid lines for one axis to show up
       },
     },
-    x: { ticks: { beginAtZero: true } },
+    x: {
+      ticks: {
+        beginAtZero: true
+      },
+      grid:{
+        color:"rgb(158,158,158, 1.0)",
+      }
+    },
+    y: {
+      ticks: {
+        display: false
+      },
+      grid:{
+        color:"rgb(158,158,158, 1.0)",
+      }
+    },
   },
 };
 
@@ -118,6 +133,7 @@ const scatterChartOptions = {
       type: "linear",
       position: "bottom",
       grid: {
+        color:"rgb(158,158,158, 1.0)",
         display: true,
         lineWidth: (context) => {
           // Make the zero line thick (3) and other grid lines thin (1)
@@ -132,6 +148,7 @@ const scatterChartOptions = {
       type: "linear",
       position: "left",
       grid: {
+        color:"rgb(158,158,158, 1.0)",
         display: true,
         lineWidth: (context) => {
           return context.tick.value === 0 ? 3 : 1;

freedata_gui/src/store/settingsStore.js

@@ -29,9 +29,9 @@ const defaultConfig = {
       maximum_bandwidth: 3000,
     },
     NETWORK: {
-            modemaddress: "127.0.0.1",
-            modemport: 5000,
-        },
+      modemaddress: "127.0.0.1",
+      modemport: 5000,
+    },
     RADIO: {
       control: "disabled",
       model_id: 0,

freedata_gui/src/styles.css

@@ -81,9 +81,6 @@ emoji-picker {
   width: 100%;
 }
 
-
-
-
 /* force gpu usage
 https://stackoverflow.com/questions/13176746/css-keyframe-animation-cpu-usage-is-high-should-it-be-this-way/13293044#13293044
 */

freedata_server/constants.py

@@ -1,9 +1,10 @@
 # Module for saving some constants
 CONFIG_ENV_VAR = 'FREEDATA_CONFIG'
 DEFAULT_CONFIG_FILE = 'config.ini'
-MODEM_VERSION = "0.17.0-alpha"
+MODEM_VERSION = "0.17.0-beta"
 API_VERSION = 3
 LICENSE = 'GPL3.0'
 DOCUMENTATION_URL = 'https://wiki.freedata.app'
 STATS_API_URL = 'https://api.freedata.app/stats.php'
-EXPLORER_API_URL = 'https://api.freedata.app/explorer.php'
+EXPLORER_API_URL = 'https://api.freedata.app/explorer.php'
+MESSAGE_SYSTEM_DATABASE_VERSION = 0

freedata_server/message_system_db_manager.py

@@ -1,14 +1,15 @@
 # database_manager.py
 import sqlite3
 
-from sqlalchemy import create_engine, text
+from sqlalchemy import create_engine, text, inspect
 from sqlalchemy.orm import scoped_session, sessionmaker
 from threading import local
-from message_system_db_model import Base, Station, Status, P2PMessage
+from message_system_db_model import Base, Config, Station, Status, P2PMessage
 import structlog
 import helpers
 import os
 import sys
+from constants import MESSAGE_SYSTEM_DATABASE_VERSION
 
 class DatabaseManager:
     """Manages database connections and operations.
@@ -274,3 +275,154 @@ class DatabaseManager:
             session.flush()  # To get the ID immediately
         return status
 
+
+    def check_database_version(self):
+        """Updates the database schema to the expected version.
+
+            This method is called by `check_database_version` if the current
+            schema version is older than the expected version. It performs
+            the necessary schema updates based on the current and expected
+            versions. Currently, it only logs a message indicating that
+            schema updates are not yet implemented. It takes the current and
+            expected versions as arguments, but doesn't use them yet.
+
+            Args:
+            current_version (int): The current database schema version.
+            expected_version (int): The expected database schema version.
+        """
+        session = self.get_thread_scoped_session()
+        try:
+            config = session.query(Config).filter_by(db_variable='database_version').first()
+            if config is None:
+                config = Config(db_variable='database_version', db_version="0")
+                session.add(config)
+                session.commit()
+                self.log("No database version found. Assuming version 0 and storing it.")
+
+            current_version = int(config.db_version)
+            expected_version = int(MESSAGE_SYSTEM_DATABASE_VERSION)
+
+            if current_version < expected_version:
+                self.log(
+                    f"Database schema outdated (current: {current_version}, expected: {expected_version}). Updating schema...")
+                self.update_database_schema()
+                config.db_version = str(expected_version)
+                session.commit()
+                self.log("Database schema updated successfully.")
+            elif current_version > expected_version:
+                self.log("Database version is newer than the expected version. Manual intervention might be required.",
+                         isWarning=True)
+            else:
+                self.log("Database schema is up-to-date.")
+        except Exception as e:
+            session.rollback()
+            self.log(f"Database version check failed: {e}", isWarning=True)
+        finally:
+            session.remove()
+
+    def update_database_schema(self):
+        """Updates the database schema to the latest version.
+
+        This method updates both tables and columns within the database
+        schema. It checks for and adds any missing tables or columns
+        defined in the model. It logs the progress and results of the
+        schema update.
+
+        Returns:
+            bool: True if the schema update was completely successful,
+            False otherwise.
+        """
+        self.log("Starting schema update...")
+        tables_success = self.update_tables_schema()
+        columns_success = self.update_columns_schema()
+
+        if tables_success and columns_success:
+            self.log("Database schema update completed successfully.")
+            return True
+        else:
+            self.log("Database schema update completed with errors.", isWarning=True)
+            return False
+
+    def update_tables_schema(self):
+        """Updates the database schema by adding new tables.
+
+        This method checks for any new tables defined in the data model
+        (Base.metadata) that are not present in the database. If new
+        tables are found, it creates them in the database. It logs the
+        names of the tables being added and any errors encountered during
+        the process.
+
+        Returns:
+            bool: True if the table update was successful, False otherwise.
+        """
+        self.log("Checking for new tables to add to the schema.")
+        success = True
+        inspector = inspect(self.engine)
+        existing_tables = inspector.get_table_names()
+
+        new_tables = [table for table in Base.metadata.sorted_tables if table.name not in existing_tables]
+
+        if new_tables:
+            table_names = ", ".join(table.name for table in new_tables)
+            self.log(f"New tables to be added: {table_names}")
+            try:
+                Base.metadata.create_all(self.engine, tables=new_tables)
+                self.log("New tables have been added successfully.")
+            except Exception as e:
+                self.log(f"Error while adding new tables: {e}", isWarning=True)
+                success = False
+        else:
+            self.log("No new tables to add.")
+        return success
+
+    def update_columns_schema(self):
+        """Updates the database schema by adding missing columns.
+
+        This method checks for any missing columns in existing tables by
+        comparing the database schema with the defined models. If missing
+        columns are found, it adds them to the respective tables using
+        ALTER TABLE statements. It handles different column types,
+        nullability, and default values. It logs the DDL statements and
+        any errors encountered during the process.
+
+        Returns:
+            bool: True if the column update was successful, False otherwise.
+        """
+
+        self.log("Checking for missing columns in existing tables.")
+        success = True
+        inspector = inspect(self.engine)
+        existing_tables = inspector.get_table_names()
+
+        for table in Base.metadata.sorted_tables:
+            if table.name in existing_tables:
+                existing_columns_info = inspector.get_columns(table.name)
+                existing_column_names = {col["name"] for col in existing_columns_info}
+
+                for column in table.columns:
+                    if column.name not in existing_column_names:
+                        col_name = column.name
+                        col_type = column.type.compile(dialect=self.engine.dialect)
+                        nullable_clause = "" if column.nullable else "NOT NULL"
+                        default_clause = ""
+                        if column.default is not None and hasattr(column.default, "arg"):
+                            default_value = column.default.arg
+                            if isinstance(default_value, str):
+                                default_clause = f" DEFAULT '{default_value}'"
+                            else:
+                                default_clause = f" DEFAULT {default_value}"
+
+                        ddl = (
+                            f"ALTER TABLE {table.name} ADD COLUMN {col_name} {col_type} "
+                            f"{nullable_clause}{default_clause}"
+                        )
+                        ddl = " ".join(ddl.split())
+                        self.log(f"Adding missing column with DDL: {ddl}")
+                        try:
+                            with self.engine.connect() as conn:
+                                conn.execute(text(ddl))
+                            self.log(f"Column '{col_name}' added to table '{table.name}'.")
+                        except Exception as e:
+                            self.log(f"Failed to add column '{col_name}' to table '{table.name}': {e}", isWarning=True)
+                            success = False
+        return success

freedata_server/server.py

@@ -258,6 +258,7 @@ def main():
     app.schedule_manager = ScheduleManager(app.MODEM_VERSION, app.config_manager, app.state_manager, app.event_manager)
     app.service_manager = service_manager.SM(app)
     app.modem_service.put("start")
+    DatabaseManager(app.event_manager).check_database_version()
     DatabaseManager(app.event_manager).initialize_default_values()
     DatabaseManager(app.event_manager).database_repair_and_cleanup()
     DatabaseManagerAttachments(app.event_manager).clean_orphaned_attachments()

tools/macOS/README.md

@@ -1,37 +1,44 @@
 ## FreeDATA Scripts for Apple macOS
 
 ### Preface
+
 The installation requires an already working MacPorts or Homebrew installation on your Mac, please follow the corresponding instrutions on https://www.macports.org/install.php or https://brew.sh  
 The scripts run on Apple Silicon. It's not tested on Intel Macs.\
 I include two short instruction how to install MacPorts or Homebrew. Please install only one of them!
 
-
 #### Short MacPorts installation instructions
+
 Install the Apple Command Line Tools\
 Open the Terminal, you find it in the Utilities Folder inside the Applications Folder, and execute the following command:
+
 ```
 % xcode-select --install
 ```
-Download the required MacPorts version from the link above and install it as usual. (Double click the pkg and follow the instructions) 
+
+Download the required MacPorts version from the link above and install it as usual. (Double click the pkg and follow the instructions)
 If you have the Terminal open, please close it completely [command+q] to make shure that the MacPorts environment is loaded.
 
-
 #### Short Homebrew installation instructions
+
 Install the Apple Command Line Tools\
 Open the Terminal, you find it in the Utilities Folder inside the Applications Folder, and execute the following command:
+
 ```
 % xcode-select --install
 ```
+
 This will take some time, depending on the speed of your mac and internet connections. After successfull installation install brew:
+
 ```
 % /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
 ```
+
 brew tells you at the end of the installation, to execute some commands. please don't forget them. Close the Terminal completely [command+q]\
 
-
-
 ### Install FreeDATA
+
 Open the Terminal and execute the following commands:
+
 ```
 % mkdir ~/freedata
 % cd ~/freedata
@@ -41,14 +48,13 @@ Open the Terminal and execute the following commands:
 % bash install-freedata-macos.sh
 ```
 
-
 ### Run FreeDATA
+
 As usual, open the Terminal and execute the following commands:
+
 ```
 $ cd ~/freedata
 $ bash run-freedata-macos.sh
 ```
+
 Your browser should open the FreeDATA webinterface. Please follow the instructions on https://wiki.freedata.app to configure FreeDATA.
-
-
-