The major new feature that's been in the works for several months and has now landed in the git "master" branch is the New FileBase:ãã_New FileBases_ããThe new FileBase files are stored in the same database location as beforeã(e.g. data/dirs/), but the file extensions are different:ãã^ Purpose ^ Old ^ New ^ã| Index (e.g. filenames) | code.ixb | code.sid |ã| Data (e.g. descriptions) | code.dat | code.shd |ã| Extended Descriptions | code.exb | code.sdt |ã| Metadata | code.dab | code.ini |ã| Allocation Tables | N/A | code.sda and code.sha |ããIf these new filebase file extensions look familiar to you, that's becauseãwe're using the Synchronet Message Base format/library (v3.0 now) for the underlying database. This means that the SMB tools you may be familiar with (e.g. CHKSMB, FIXSMB, SMBUTIL) also work on the new filebases.ããThe conversion of the filebases to the new format should occur automatically when you run 'jsexec update' which in turn will execute the program 'upgrade_to_v319' when appropriate (just one time). Once converted, you can delete the old files or leave them in place in case you need to revert back to Synchronet v3.18 for any reason. The old filebase files won't harm anything if left.ããThe creation of each new filebase will automatically calculate and store the hashes of the contents of the actual files available for download. Theseãhashes are useful for duplicate file detection and data integrity assurance.ãIf you wish to opt-out of the file hashing (which consumes the majority of the time during the upgrade process), you can turn off file hashing in the per-directory Toggle Options in SCFG->File Areas. You would have to performãthis opt-out for the directories of choice *before* you run 'jsexec update' / 'upgrade_to_v319'. You should not normally need to run 'upgrade_to_v319'.ãã_Long Filenames_ããWhile filenames stored in the filebases used to be limited to MS-DOSãcompatible 8.3 formatted names, longer filenames are now supported on all platforms. Additionally, some previously invalid filename characters (e.g. spaces) are now allowed and files without extensions (i.e. no '.' in their filename are now supported).ããAlthough Synchronet for Windows previously used Win32 API functions for short <-> long filename conversions in the Windows builds of Synchronet, resultingãin the unfortunate Micros~1 shorted filenames stored and sometimes seen, thatãis no longer the case. Except for the %~ command-line specifier, those short/long filename conversion functions are no longer in use anywhere within Synchronet for Windows - it's native filenames through-out. The filebase conversion process (upgrade_to_v319 / 'jsexec update') on Windows will attempt to automatically resolve the native/long filenames and store those names and only those names in the new filebases.ããNote: abbreviated versions of long filenames are displayed in some situationsãto accommodate the limited width of a traditional BBS user terminal. An effort is made to always display the full file extension/suffix howeverã(e.g. "longfilename.jpeg" may be *displayed* as "longfil.jpeg").ããNote: only 64 characters of each filename (always including any extension) are indexed for searches and duplicate checking, but the entire filename, up toã64K characters in length, is stored intact in the filebase.ããFilenames with /extensions/ longer than 3 characters, e.g. ".jpeg", ".tar.gz", can be added to the filebases, but the configurable compressible, extractable, and viewable file types/extensions remain limited to 3 characters in SCFG. Similarly, a maximum length of 3 character archive "types" are stored per BBS user record (for each user's QWK packet format and temp archive preference).ã_Large Files_ããFiles greater in size than 2GB or 4GB (depending) were previously a problem. Though there are still some 32-bit file length limitations (e.g. only files smaller than 4GB in size will be hashed), there is better and increasingãsupport for larger files in general.ããNote: the ZMODEM transfer protocol as designed by Chuck Forsberg only supports files up to 4GB in size and in many cases, files greater than or equal to 2GBãin size will prove difficult or impossible to transfer between some ZMODEM implementations. In general, it is recommended to use an alternate transfer protocol (e.g. YMODEM[-G], FTP, HTTP) for files >= 2GB in length.ãã_Large Directories_ããDue to the new filebase design, directories with more than 10,000 files areãnow supported (though, not encouraged).ãã_Descriptions_ããThe file "summary" or single-line "short description" remains limited to 58 characters in length for practical purposes. Though longer file summariesã(up to 64KB) can be stored in the filebase, they are not recommended.ããExtended (multi-line) descriptions may now span more than the previous limitãof 10 lines or 512 total characters. There is no technical limit to the length of extended file descriptions, though a limit of 1024 characters imported from description files embedded in archives (e.g. FILE_ID.DIZ) is being imposed. If you have a need for longer (than 1024 character) extended descriptionsãimported from embedded description files, please provide me with details.ã_Batches_ããFile upload and download batch queues used to be maintained in memory (though they were written to disk files to be retained between user logons), they are now entirely maintained in disk files (data/user/*.upload and *.dnload in .ini file format). This means that custom batch management can now be performed easily by modules or non-Terminal Server scripts.ãã_Hashes_ããFiles are now hashed, by default, using multiple hashing algorithms (CRC16, CRC32, MD5, and SHA1) for duplicate file detection and for reporting to users (e.g. to insure data integrity). For a file to be considered a duplicateã(i.e. and rejected for upload) it must have the same size and hash values as another file in a filebase already. Each directory is configurable as toãwhether or not to hash its files or use it for duplicate file detectionã(by name or hash).ãã_Sorting_ããThe new filebases are indexed in the order in which the files are importedãinto the database. Sorting of the files for display purposes in the terminalãand FTP servers is optional and configured by the sysop:ã Name Ascending (case-insensitive)ã Name Descending (case-insensitive)ã Name Ascending (case-sensitive)ã Name Descending (case-sensitive)ã Date Ascendingã Date DescendingããAs a result, the "RESORT" file transfer operator command has been removed.ã_Tags_ããIndividual files can now be tagged for easy searching/grouping. This feature will be utilized/enhanced more in the future.ãã_JavaScript_ããThe new "FileBase" class is used (similar to the existing MsgBase class) toãopen and access filebases from JavaScript modules. Using this class, the defaults methods of listing and transferring files can be replaced withãcustom modules.ãã_TickIT_ããThe new FileBase JS class is now used to import files directly from FidoNet -style .TIC files (via tickit.js) so no dependency or invocation of anyãexternal utilities (e.g. addfiles) is required.ãã_Utilities_ããThe native utilities ADDFILES, FILELIST, DELFILES, and DUPEFIND have been replaced with similarly named and purposed JavaScript utility scripts to be invoked with JSexec:ã- addfiles.js for importing lists of files into filebasesã- postfile.js for importing a single file into a filebaseã- filelist.js for generating file listings from filebasesã- delfiles.js for removing files from filebasesã- dupefind.js for discovering and reporting duplicate files in the filebasesã_Performance_ããDue to the nature and use of the new filebase API, file listings are muchãfaster (e.g. large file listings from the Synchronet FTP server) as well as filename/pattern, description text, and duplicate-file searching.ãã_FTP Server_ããDue to the removal of support for rendering FTP-downloaded content (e.g.ãHTML files) in modern web browsers, the FTP Server no longer supports dynamic HTML index file generation (e.g. 00index.html). Instead, we will focus onãbetter support for filebase browsing and file transfers via HTTP and HTTPS in addition to the traditional FTP and FTPS uses. The dynamic generation ofãthe ASCII file listings via FTP (e.g. 00index) is still supported by the FTP server, though now much faster than before.ãã_libarchive_ããThe libarchive library (http://libarchive.org/) has now been integrated into Synchronet (and exposed via the new "Archive" JavaScript class) and integrated into SBBSecho so that the creation, listing/viewing, and extraction ofãarchived files can now be performed "in-process" without the invocation of or dependency on external programs (e.g. Info-Zip unzip or PKUNZIP).ããFormats fully supported:ã- zipã- 7zipã- gzipped-tarã- bzipped-tarããFormats supported for viewing and extraction only:ã- rarã- lha/lzhã- isoã- xarã- cabããThis means that for most BBSes, no "Compressible" or "Extractable" file types need to be configured in SCFG->File Options. Additionally, by settingã"Archive Format" to "ZIP" for SCFG->Networks->QWK->Hubs, no "pack" or "unpack" command-line need be configured.ããFor listing the contents of archives, the new archive.js utility script may be installed as a "Viewable File Type" handler for the commonly supported file extensions by running 'jsexec archive.js install'.ããIf you're building sbbs on *nix, you'll need the libarchive development package (e.g. libarchive-dev) installed or you will get build errors.ã_DIZ_ããDescription files embedded in archives (e.g. FILE_ID.DIZ) are now supportedãmore uniformly and seamlessly.ãã_File Echoes_ããEach file transfer directory configured in SCFG->File Areas may now have an "Area Tag" explicitly set for FidoNet-style file distribution networks. Ifãan Area Tag is not explicitly set, then the directory's short name is usedã(with spaces replaced with underscores) automatically. tickit.js now usesãthis new "area_tag" file_area.dir[] JS property for its "AutoAreas" feature.ã_User to User Files_ããThe user to user file transfer feature has been removed. Send file attachments with email/netmail if you want to send files to users.ãã_Opened Files_ããThe "open" (reference) counter for files is now gone. If you want to removeãa file from the filebase while a user has it in their batch download queue orãis actively downloading it, nothing is preventing you from doing so.ããAs a result, the "CLOSE" file transfer operator command has been removed.ã_Alternate File Paths_ããThe support for "Alternate File Paths" has been removed. There are betterãmodern operating/file system solutions to the original problem solved withãthis feature.ããAs a result, the "ALTUL" file transfer operator command has been removed.ã_Bi-directional File Transfers_ããThe protocol drivers that supported bi-directional file transfers (Bi-Modem, HS/Link) are now long unsupported DOS/OS2 programs with no equivalent in the modern world. Bye bye Bi-modem. :-(ã--ã digital manããSling Blade quote #10:ãMorris: I stand on the hill, not for thrill, but for the breath of a fresh kill Norco, CA WX: 55.5øF, 71.0% humidity, 0 mph SSW wind, 0.00 inches rain/24hrsã---ã þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.netã