0000888: Support an application transparent file conversion, compression and encryption solution like z/OS-Subsys for open systems

Notes
(0001114) Falk Reichbott (administrator) 2017-08-29 11:04	FUSE on AIX https://www.ibm.com/developerworks/aix/library/au-spunix_newfs/index.html [^]

(0001116) Falk Reichbott (administrator) 2017-08-31 09:23 edited on: 2017-08-31 12:23	Solution with FUSE We implement a main (executable) for mount 'flucfuse' with standard fuse parameter (only write or read and write, only user or root or other, in forground or background, ...) and one optional file name with the write definitions in the CLP syntax below: fluc.write(file='matchstringwithwildcards' substitute[(format="format.string" file="write/output.string" [readable])] where substitute is an array for multiple writes (we use n times fcbopen() and do n times fcbpwrite() for each handle) and only one of them can have the readable flag set. If no central config file name is given, a hidden file (.flucfuse.write) is used in each directory. New write definitions might be added on the fly. If the readable flag is defined, then the inverse format (must be implemented) and read (must be changed from pfIvr to pdIvr) strings are written to a hidden file (.flucfuse.read) in each directory with the CLP syntax below: fluc.read(file='logicalfilename' substitute(format="format.string" file="read/input.string") attributes(filesize=4711)) This file is used for read requests in inverse form. The filesize for stat and other attributes are also stored in this directory extension. Additionally we must finalise the chmod support, so that the user can define the access rights for the physical files in the "write/output.string". The FLUC byte interface (FLCBYT) must be extended by the IVR functions to get the inverse strings for the read operation. If multiple FIOs are defined in the "output.string" then only the first FIO definition must be used to build the inverse command, the other inverse strings must be comments, so that this implicit rule makes the read unique. The byte interface gets 3 new function. First fcbpread() and fcbpwrite() which manage the offset on the FCB handle. The third will be an extension to the close function which provides the attribute string for the logical file. At each close of a converted file, the inverse entry is made if the readable flag is set and the read specification file is written. Additionally all other FCB functions get a thread save version with errno, trcsiz and trcbuf as the first 3 parameter. By linking the flucfuse main programm the LMAP must be checked to verfiy that no static data area is present, in the implementation of the FLUC file system. If a file don't fit with the match string a passthrough will be done. If a file matches, then the locical name is given in the state string to fcbopen() and will be converted to the pysiscal name based on the defined mapping rules in the "write/output.string". The physical name is given back in the inverse read string. To automate reload of a central config file inotify should be used. Changes to config files within the work tree are seen anyway and can be used instantly. The implementation is limited to sequential read and write. If an application does seek and makes updates, an I/O error is flagged. A special case is a seek_end, ftell and rewind. This cases are covered by a complete read (if possible) and a reopen of the file afterwards. There will be another filesystem called flucfusecheck which makes passthrough, and logs per file all functions and check the offsets for sequential read and write. This file system can be mounted to figure out, which files of this mountpoint can be converted by a write specification and which files will fail. Questions: Is a central config file for write operations useful? The implementation would be more simple if the configuration is done as an extension to each directory only based on the file name, without any path. For example you mount the flucfuse on directory pgpfile in your home directory and you make the definition in the file pgpfile/.flucfuse.write. fluc.write(file='*' substitute(format="format.bin()" file="write.bin(encr.pgp(user='~'))" readable) In this case each file of this folder will be OpenPGP encrypted under the own user id. Another solution could be that the config file at mount defines the conversion for the whole mountpoint. In this case there is no passthrough implemented. All files under this mount point are converted as defined at mount time. This requires a mount per write definition. Would this be more useful?

(0001117) Falk Reichbott (administrator) 2017-08-31 10:23 edited on: 2017-08-31 12:26	For the read specification it makes more sense to work with a hidden directory where for each converted file a hidden file is stored with the inverse read specification. /home/user/OpenPGPFiles/.flucfuse.write /home/user/OpenPGPFiles/.flucfuse.read/.myfirstfile.txt.flucfuse.read /home/user/OpenPGPFiles/.flucfuse.read/.mysecondfile.txt.flucfuse.read /home/user/OpenPGPFiles/myfirstfile.txt.pgp /home/user/OpenPGPFiles/mysecondfile.txt.pgp This makes the access to the read specification easier.

(0001119) Falk Reichbott (administrator) 2017-08-31 16:56 edited on: 2017-09-01 10:01	FUSE on AIX is not possible based on missing kernel support for it. IBM recommend to use NFS on AIX in conjuction with a LINUX server where FUSE exists.

(0001120) Falk Reichbott (administrator) 2017-08-31 17:06	A reposition to a previos location at read, can be simply implemented, see below: if (uiOffset<psHdl->uiOffset) { psHdl=fcbreopen(psHdl);//makes close and open fcbpos(psHdl,uiOffset); } else if (uiOffset>psHdl->uiOffset) { fcbpos(psHdl,uiOffset); } fcbread(psHdl,uiLen,pcDat); The function fcbreopen and fcbpos must be implemented.

(0001121) Falk Reichbott (administrator) 2017-09-01 13:02 edited on: 2017-09-08 09:33	An additional hidden file containing the last read or write error must be added to the hidden folder described above. /home/user/OpenPGPFiles/.flucfuse.read/.myfirstfile.txt.flucfuse.read.errortrace /home/user/OpenPGPFiles/.flucfuse.read/.myfirstfile.txt.flucfuse.write.errortrace

(0001131) Falk Reichbott (administrator) 2017-11-05 10:18 edited on: 2017-11-08 15:12	The internal flucFSgz (transparent compression of all files in the folder to GZIP files) works fine, now we start with the dynamisation of the conversion capabilities (making the write string variable) and the selection stuff (wildcard pattern matching) for the files.

(0001151) Falk Reichbott (administrator) 2018-03-08 18:47	flucFS is using FUSE to mount an existing directory in some other folder. Everything written to the mounted folder is transparently converted, encrypted, compressed as defined in the configuration. If no match is found in its configuration, the data is passed through without any change. The configuration is done with the argument strings of the FLUC byte interface API. However, since flucFS is a filesystem and uses only the read and write functions of the API, the element access features of the API are not useable with flucFS. flucFS2 provides the same functionality as flucFS but uses the older version 2 of FUSE. This should only be used on systems without support of FUSE version 3. flucFSconf is a command line utility to view and modify the configuration of flucFS. A fast memory mapped database is used to store configuration and meta data. This database is stored in a subdirectory of fluc-root with the name .flucIndex. flucFSconf will check the formatString and fileString arguments for correct syntax before storing them. On error the exit code is set accordingly and a detailed error message is printed on stderr. flucFSgui graphical user interface to view and modify flucFS configuration. The fuse kernel module must be loaded in order to use any fuse filesystem. This kernel module is usually installed with the linux kernel.

Issue History
Date Modified	Username	Field	Change
2017-08-29 10:53	Falk Reichbott	New Issue
2017-08-29 10:53	Falk Reichbott	Status	new => assigned
2017-08-29 10:53	Falk Reichbott	Assigned To	=> Ulrich Schwab
2017-08-29 11:04	Falk Reichbott	Note Added: 0001114
2017-08-31 09:23	Falk Reichbott	Note Added: 0001116
2017-08-31 10:23	Falk Reichbott	Note Added: 0001117
2017-08-31 12:23	Ulrich Schwab	Note Edited: 0001116	View Revisions
2017-08-31 12:26	Ulrich Schwab	Note Edited: 0001117	View Revisions
2017-08-31 16:56	Falk Reichbott	Note Added: 0001119
2017-08-31 17:06	Falk Reichbott	Note Added: 0001120
2017-09-01 10:01	Ulrich Schwab	Note Edited: 0001119	View Revisions
2017-09-01 13:02	Falk Reichbott	Note Added: 0001121
2017-09-07 14:31	Falk Reichbott	Description Updated	View Revisions
2017-09-08 09:33	Ulrich Schwab	Note Edited: 0001121	View Revisions
2017-10-23 12:53	Falk Reichbott	Relationship added	related to 0000891
2017-11-05 10:18	Falk Reichbott	Note Added: 0001131
2017-11-08 15:12	Ulrich Schwab	Note Edited: 0001131	View Revisions
2017-11-10 09:36	Falk Reichbott	Relationship added	related to 0000704
2018-03-06 11:29	Ulrich Schwab	Description Updated	View Revisions
2018-03-06 11:29	Ulrich Schwab	Additional Information Updated	View Revisions
2018-03-08 18:47	Falk Reichbott	Note Added: 0001151
2018-03-08 18:47	Falk Reichbott	Status	assigned => resolved
2018-03-08 18:47	Falk Reichbott	Fixed in Version	=> 5.1.17
2018-03-08 18:47	Falk Reichbott	Resolution	open => fixed
2018-03-08 18:48	Falk Reichbott	Relationship added	related to 0000912