Computer Storage Planning
Written by Dave Barth, January 15, 2005
PREFACE
When a new project is launched, there is usually little thought or planning given to designing a file structure and naming convention that will serve the new system
throughout its life cycle. The result of limited planning for file creation, retention, and deletion is that the amount of space allocated to the system is filled up over a
period of time, necessitating a reactive initiative to delete obsolete files in order to free space.
FILE STORAGE COSTS
Although the cost per byte of storage continues to drop, major systems use a lot of space, sometimes costing millions of dollars per year, and the tendency is that
the space requirement for a system with an unregulated storage philosophy gradually increases, unchecked, toward infinity.
FILE KNOWLEDGE
The longer a system is in place, the knowledge of why a file was created and when it is no longer useful is lost. Many developers attach an unlimited file retention
parameter to files, which causes the operating system to retain them forever. When persons leave the project or forget about a limited-life test file they have created,
the existence of the file becomes problematic because no one knows if the file needs to be saved or if it can be deleted.
AUTOMATED ARCHIVING
For files that need to be retained, but are not used very often, mainframe operating systems have an automatic archival capability, but to use it effectively, the file
structure and naming convention is critical.
UNCONTROLLED FILE CREATION AND DELETION
When storage becomes tight due to uncontrolled creation of files, a reactive, ad hoc project is usually instituted to try to delete old, unneeded files. This process
is usually hindered by the lack of knowledge about why the files were created and if they have any current usefulness to the system. As often happens, files that
are rarely needed are deleted because there is no evidence that they are required by the system. If a rarely needed function that uses them is eventually initiated,
it fails because the file is no longer available. When file creation is uncontrolled, the result is usually uncontrolled file deletion.
RETROACTIVE FILE CONTROL CHANGE
An adjunct project to the deletion effort is to change the way files are allocated and retained. Although this is often difficult and expensive to achieve, it can help
alleviate insufficient space problems in the future. However, funding is usually lacking to carry this project through to completion. Once the ad hoc file deletion
effort has freed up space, the need to control file creation and retention is deemed a low priority and is forgotten until space is used up, once again, and the
cycle of frantic file deletion starts again.
LESSONS LEARNED
When a new project is established, it is critical to design a file structure and naming convention to provide file control within the capabilities of the hardware and
operating system being used. Test files with a limited useful life should be created with a realistic retention period so that they will be automatically deleted when
the expiration time occurs, if the operating system has that ability. Creation of datasets with unlimited retention should be reserved for system-critical files. Each
person who creates files should identify them with a node code to show ownership and place them in a temporary area set aside for limited-life files. System-critical
files should be stored in their own, protected area. Although implementation of these concepts might not be feasible in all cases, at least they should be discussed
and a methodology hammered out in the beginning to preclude frantic storage recovery operations when the available space is used up. This approach will also
support CMMI Level 5 precepts.