X-Git-Url: http://dxcluster.org/gitweb/gitweb.cgi?a=blobdiff_plain;f=Changes;h=840dd943e459b5a0d66a4dbcfe909d35806e5e94;hb=refs%2Fheads%2Fcachespots;hp=1563721c937b60775ca49bc2e371922300bc68f2;hpb=d6f642282a0dfa2e55b62bd792ebe8b41192356c;p=spider.git diff --git a/Changes b/Changes index 1563721c..840dd943 100644 --- a/Changes +++ b/Changes @@ -1,3 +1,277 @@ +18Nov21======================================================================= +1. Add a dx spot cache which will store 2 days worth (configurable) of spots. + This is then used by the vast majority of simple sh/dx queries without + using a sub process. A simple sh/dx is any sh/dx that does not have an + "expression" such as "sh/dx g1tlh". So "sh/dx", "sh/dx 200", "sh/mydx" will + always run in the main program via the cache. Certain client programs (no + names, no pack drill) will notice the difference immediately. It's also a + bit snappier. +2. set/badspotter or set/badnode will work for RBN input. +3. RBN input with invalid QRGs will be dropped with a nice friendly message + rather than a scary exception message about multiplication thrown by one + of the libraries that I use. +4. Added CTY-3126 prefixes. +5. Silently ignore self spotting below 1240000Khz (change + $Spot::minselfspot to 0 to disable or some other freq in Khz). NOTE this + will appear as a spot to the spotter's feed, but will not be passed on. +15Jul21======================================================================= +1. Fix long standing bug in crontabs that commands with " rather than ' around + strings could cause those commands not to work. +2. Add CTY 3117 prefixes. +15Mar21======================================================================= +1. Fix DXUtil::localdata so that it does actually prefer the latest version + of a file regardless of whether it is in /spider/data or /spider/localdata. +2. Add CTY-3105 prefixes. +05Mar21======================================================================= +1. Fix DXCron::spawn_cmd so that more than one spawn_cmd can be active at a + time. This affects many nodes where lots of regular spawn_cmds happen at + a time and this prevented crucial things like 'start_connect' from working! +2. Sort user lockout issues to cope with all likely scenarios - including + "phantom" SSID working. +3. Fix "close_gracefully" error messages which are useless, but benign. These + happen (mainly) when outgoing connects timeout. +24Feb21======================================================================= +1. modify RBN timing arrangements + + There are two new twiddle pots: + + $limbotime (default 5*60 secs) which allows potential spots to + hang around longer to allow $minqual *different* skimmers to spot + them. At which point they are emitted. This is to catch the "slow + burning" spots where a call is spotted by different skimmers but + separated in time by more than $dwelltime - in fact maybe by + minutes. + + $maxqual (default 9 skimmers) which short circuits the normal + hard maximum $quality (9 skimmer spots) and $dwelltime (10secs) + to allow a spot that has $maxqual *different* skimmers (usually + as the result of a burst from the RBN) to defeat $dwelltime and + be emitted the moment that condition is satisfied. There maybe + a better name for this. +21Feb21======================================================================= +1. add data section on 160m in bands.pl. +07Sep20======================================================================= +1, Change interface to watchdbg & grepdbg slightly so that multiple search + regexes are ANDed rather than ORed together. ORing is easily achieved + already by the usual regex pattern 'PATT..|PATT..|..' whereas ANDing could + not be done as easily without resorting lots of 'PATT.*PATH' things which + would not necessarily get what was wanted. +2. Make sure that the pc92 C record only contains nodes and users and not + other extranoeus things like skimmers... +15Aug20======================================================================= +1. Simplify the skimmer scoring mechanism. +13Aug20======================================================================= +1. Improve the (displayed) RBN frequency weighting the skimmers' frequencies + w.r.t majority view on each spot. Any skimmer that disagrees with a + gets docked a 'point' from their 'good' score (if any) and have a 'point' + added to their 'bad' score. This score is then taked into account during + the 'choosing the real frequency' process. If a skimmer agrees with the + majority the process is reversed. To see this in action: set/deb rbnskim +06Aug20======================================================================= +1, Add CTY-3013 Prefixes +2. Make RBN more efficient. Start the process of skimmer node performance + caching. Add minimum quality allowed (at 2), which will be overrideable. + The format of the rbn_cache has changed and so a full restart will occur. +3. Collect channel input/output stats. New command: show/data_stats to show + them. +4. Add local::lib qw{/spider/perl5lib} to store cpanm loaded modules just for + DXSpider. This is done so that updates needed by future changes can be done + as the sysop user and doesn't have to be done as root. This paves the way + for UPDATE.pl which will pull down new modules that it needs automatically. + When it's written, which will be soon. +29Jul20======================================================================= +1. Add show/rbn command that allows one to see who is online and configured + for RBN. See help sh/rbn for details. +2. Fixed issue with set/ve7cc mode adding unwanted newlines to output. +3. Attempted to make the QRG normalisation more "intelligent". This would be + sooooooooo much easier if skimmers were calibrated more accurately. +4. Added CTY-3012 prefixes. +23Jul20======================================================================= +1. fix Filtering so that it does less of what it was doing badly, thus + the perl interpreter to do more of the work. Which means you *should* be + able to define more or less any allowable operation for that type of + filter. Which is a complicated way of saying: you can now use brackets. +2. Regex expressions now appear as readable text in sh/filter. +3. Make the RBN status file JSON, rather than DD format. In future versions + it will also be VERSIONed to enable more features to be added + transparently. +4. Add show/rbn command to allow sysops to see who is using rbn. +11Jul20======================================================================= +1. Fix (embarrassing) gratuitous '#'s in bands.pl +2. IMPORTANT: certain PC92 strings can crash DXSpider if the nologchan debug + category is set (which reduces debug files to sizes that non-developers can + reasonably store (especially during big contests)), +10Jul20======================================================================= +1. Fix console.pl permissions problem when running as another user not in + debug mode. Console.pl WILL NOT WORK in debug mode unless it is running + as the sysop user. +08Jul20======================================================================= +1. "Finish" the RBN system :-) +2. This includes enabling the coarse selection of spot modes using set/wantrbn + with arguments like 'set/wantrbn cw beacon'. This limits your output to + just CW, BCN and DXF modes. +3. The RBN spot is now cached. With a following wind, this means that even a + node restart, done in a timely fashion (within a few minutes) will not + cause a "cache warmup" delay for users on a restart. +4. Added the "full fat" set/wantrbn command and aliased it to 'set/skimmer'. + I use both terms (whenever I remembered) in the help text. +5. Help text has been written. +6. The UPGRADE.mojo file has been tweeked to point out the users file format + change. +7. Merge in users.v3j to the mojo branch. +8. Add CTY-3011 prefixes. +07Jul20======================================================================= +1. Fix show/node command. +2. Fix show/cluster command to take into account the presence of skimmer nodes + which are a new category of thing which is neither a node nor a user. +06Jul20======================================================================= +1. Add RBN.mojo with information of the RBN capabilities of DXSpider. +05Jul20======================================================================= +1. Fix show/dxcc. +2. Add HAPROXY "real ip" type 1 handling for incoming connections. +04Jul20======================================================================= +1. Give console.pl (or dx) a good going over with a bog brush to *finally* + (cough) make it work correctly with a full 80 column window (and not just + to a width of 79 really). Also fix scrolling. +28Jun20======================================================================= +1. Merge mojo with users.v3j to remove all vestages of Storable from DXSpider + in an effort to make the whole storage thing more reliable (and also a + bit faster). The user file will be auto-upgraded on restart. This may take + up to 20 seconds on slower hardware (and maybe a bit longer on huge user + files). On my 180,000 odd users, on my hardware, it takes 4 seconds. +2. The DXQSL system storage is also upgraded, Please run + /spider/perl/create_dxqsl.pl in a spare shell. This will recreate the + dxqsl.v1j file. Run 'load/dxqsl' in the console to activate it. +17Jun20======================================================================= +1. Change the Spot file reading mechanism back to the default of using 'tac'. +08Jun20======================================================================= +1. Fix show/mydx (lack of) filtering bug. +2. Add qra locator to prefix_data.pl. +3. Add 4 digit qra square for spotted callsign if show/dxgrid is enabled. +4. Fix general filtering bug where erroneous input causes crashing. +03Jun20======================================================================= +1. Make sure that all possible regexes get passed across to the search engine. +2. Fix out of order logging on sh/log queries spanning more than one month. +3. Do not read backwards on sh/dx, reading forwards seems a 1/3 quicker. +4. Add , and shortcuts to sh/dx (e.g sh/dx on 2m). +02Jun20======================================================================= +1. Fix the small whoopsie in sh/dx. +01Jun20======================================================================= +1. Fix sh/dx iota and qra. This completes the conversion of sh/dx's limited + "old style" parsing to using Filter style parsing with something that + allows expressions include brackets ( and ) as well as the 'not' keyword. + NOTE: the precedence rules are the same as perl's '!', '||' and '&&' + operators. +31May20======================================================================= +1. Improve links command layout slightly. +2. Issue an *accurate* UPGRADE.mojo with all the new packages included! +3. Fix issue with sh/dx +30May20======================================================================= +1. Fix sh/dx! It appears that there is a long standing problem with sh/dx not + actually return all the answers it should. I have also had a request to + allow / enable the "not" keyword (just like the filtering system dones). + + Now this was one of the earliest modules that I wrote and it took quite + a bit of work to tease out the important bits and then use the Filter + module (as in acc/spot) to generate the sh/dx filtering expressions. This + seems to have been done. + + NB This will likely not work if you are using a SQL backend to do sh/dx + commands. This was never actually an official feature. It is now + deprecated. +29May20======================================================================= +1. Please install Math::Round and Data::Structure::Util. Instructions are + available in UPGRADE.mojo. +2. ** Withdrawn ** user file format change to json not yet committed. +20May20======================================================================= +1. Backport convert-users-v3-to-v4.pl to allow creation of the new json + formatted *text* based user file from the old v3 DB_File and Storable + version. This can be done either online (with the node running) or offline + with the node stopped. This is in preparation for the next update of the + mojo branch. Running this program just *BEFORE* doing your next update of + the mojo branch *should* ensure a seemless transition to the Storable & + DB_File free version of the users file. +2. Show git branch in show/version. +3. Add CTY-3010 changes. +17May20======================================================================= +1. Backport DXSubprocess to change serialisations. + Currently the internals of Mojo::IOLoop::Subprocess defaults to + using Storeable as its cross-process argument and data serialisaion + method. It can use others. This update reverts back to the + original ForkCall method of using JSON. +10May20======================================================================= +1. Added basic changes so that users *could* have multiple connections to the + same node if it is allowed. This is work in progress and is there to see + if it deals with some networking problems encountered on very high volume + sites which can see more than 1000 users (although the problems can occur + with many hundreds or by the practices of certain ISPs). More information + will be forthcoming if I get to the bottom of what's REALLY going on and + whether this is (or maybe just one part of) the solution. It won't be + ready for general use until then. +2. Fixed the script import error reported by Joaquin (EA3CV?), by the simple + expedient of restoring the version from the master branch. +09May20======================================================================= +1. Show the route by which this PCxx came in progress debugging reports. In + some other words: the spot/wwv/wcy/ann message arrived first from this + connection. +2. Improve progress WWV & WCY messages. +3. Stop random node isolations (at least in one place) +08May20======================================================================= +1. Tidy up routing table +2. Add new argument to show/version (ALL or list of regexes) that allow you + to see the version and build nos of all nodes on the system. +07May20======================================================================= +1. Revert changes made since 25Apr concerning IP address reconciliation. +25Apr20======================================================================= +1. Add maximum no of users on node to show/cluster. +2. Add ability to show last n lines of debugging ring buffer. +3. Remove redundant wpxloc.dat file. +24Apr20======================================================================= +1. Add 'progress' debugging for showing that stuff is happening in nologchan + a.k.a ringbuffer only mode. +2. Fix grepdbg so that no regex argument simply lists the file. +3. Add CTY-3008 prefixes. +22Apr20======================================================================= +1. Fix the module search path in update_sysop.pl +2. Add latest prefixes CTY-3007 +3. Fix unset/startup command. This will only work for users' start up scripts + things like user_default or startup must be edited or removed by hand. +21Apr20======================================================================= +1. Finally fix the "actually tranmit" any output (like from 'logout' files) + to users before disconnecting. This also means that disconnecting nodes + now receive the (totally redundant, but hey) PC39 reason for disconnection. +20Apr20======================================================================= +1. Speed up sh/log (including chat, rcmd, ann etc) that search the system + log files, by removing a completely redundant subsystem and also leveraging + the core operating system utility 'tac', if it's available. +19Apr20======================================================================= +1. The long haul that is the mojo branch has started up again. It's something + to do during this lockdown. +2. The problems and instabilities around running commands that worked in + forked processes, when used other than by normally logged in users (e.g. + in the local crontab), have been fixed. +3. If a /spider/local_data/logout file is present then its contents will be + sent to the user on logout. +4. Add the ability to sh/dx origin or ip (address). +5. Retire the use of the deprecated Mojo::IOLoop::ForkCall in favour of using + the supported equivalent in Mojolicious 7.26 and above. Which means you + will need to upgrade Mojo to at least the version. The current version is + 8.36 and is known to work. +10Sep19======================================================================= +1. Improve DXSql database filtering to exclude most via type + reports. +2. Add CTY-2913 prefixes + wpxloc.raw +14Jul18======================================================================= +1. Add CTY-2808 prefixes + wpxloc.raw +16Jun18======================================================================= +1. add more modes to rbn.pl +23Jan18======================================================================= +1. Add CTY-2802 prefixes +27Oct17======================================================================= +1. Get correct hostname support and privilege levels on web connects. +26Oct17======================================================================= +1. Start (serious) work on web interface. Make the necessary changes to allow + a local webserver to connect and get its own style of messages. 11Aug17======================================================================= 1. Add default systemd service file file 10Aug17=======================================================================