The Data Handbook

I introduction.- Five Researchers and Their Data.- Judy ReSyrch—Fan Simulation.- Michael Astroe—FITS Data.- Dr. Tim Boans—MRI Scans.- Jeanne Beeker—Solubility Experiments.- Wolfram Herth—Ground Water Information.- Important Questions to Answer.- How Are the Numbers Stored?.- How Is the Data Organized?.- What Is the Dimensionality of the Data.- Is the Data on a Grid?.- What Is the Best Way to Analyze the Data?.- II Numbers in Computers.- 1. An Overview of Numbers in Computers.- Turning Data into Numbers.- Storing Numbers on Computers.- Binary Numbers and ASCII Text Numbers.- Evaluating Number Formats.- 2. Byte Numbers.- Bits and Bytes.- Unsigned Bytes.- Signed Bytes.- Twos-Complement vs. Ones-Complement.- Hexadecimal Representation.- Octal Numbers.- Kilobytes and Megabytes.- Byte Numbers: Advantages and Pitfalls.- Computational Resources.- Range of Values.- Numerical Precision.- Calculations with Bytes.- Portability.- Summary.- 3. Integers and Fixed-Point Numbers.- Short Integers and Long Integers.- Fixed-Point Numbers.- Range of Values.- Numerical Precision.- Calculations.- Truncation Error.- Calculations: The Right Way.- Byte Order.- Integers and Fixed-Point: Advantages and Pitfalls.- Computational Resources.- Range of Values.- Numerical Precision.- Calculations with Integers.- Portability.- Summary.- 4. Floating-Point Numbers.- Single-Precision Floating-Point.- Binary Fractional Numbers.- Decoding a Floating-Point Number.- Normalized Numbers.- VAX Floating-Point.- Single-Precision Range of Values.- Numerical Precision of Single-Precision Floats.- Problems with Floating-Point Calculations.- The Mystery of the Vanishing Significant Digits.- The Case of Almost-Equal Numbers.- The Trouble with Floating-Point Comparisons.- A Bevy of Roundoff Errors.- The Riddle of the Stairstep Graph.- Special Floating-Point Values.- Zero.- NaN.- INF.- Denormalized Numbers.- Additional Floating-Point Formats.- Double-Precision Floating-Point Numbers.- Complex Numbers.- Extended Formats.- Floating-Point: Advantages and Pitfalls.- Computational Resources.- FPUs.- MIPS and MFLOPS.- Range of Values.- Numerical Precision.- Calculations with Floating-Point Numbers.- Portability.- Summary.- 5. ASCII Text Numbers.- ASCII Characters.- Control Characters.- Foreign Languages.- ASCII Text Strings.- ASCII Numbers.- Exponential Notation ASCII Numbers.- Separating Numbers—ASCII Delimiters.- Space, Tab, and Comma as Delimiters.- Storing a Position but Not a Number.- Fixed Format Delimited Files.- Delimited Files—Summary.- ASCII Line Separators.- Additional Binary vs. ASCII Text Issues.- Binary vs. ASCII Text File Transfers.- Character Encoded Binary.- Binary Coded Decimal.- NFS Mounts.- ASCII Text: Advantages and Pitfalls.- Computational Resources.- Range of Values.- Numerical Precision.- Calculations with ASCII Text.- Portability.- Summary.- III The Data Universe.- 6. A Map of the Data Universe.- Data Values, Locations, and Descriptions.- Defining Data Location.- Independent Variables and Dependent Variables.- U.S. Weather Example.- The Dimensionality of Data.- Adding a Dimension.- Organizing Data Storage.- Column Data.- 2D Matrix Data.- 3D Matrix Data.- Polygonal Data.- Summary.- 7. Column Data.- Column Datafiles on Disk.- Text String Columns.- Visualizing Column Data.- Linegraphs and Bar Charts.- Scatter Plots and Parametric Plots.- Parametric Plots vs. Linegraphs.- 3D Scatter Plots.- Beyond Three Dimensions.- An Observation on Good Graphs.- Summary.- 8. 2D Matrix Data.- 2D Matrix Datafiles on Disk.- An Example of a 2D Matrix Datafile.- ASCII Text and Binary 2D Matrix Datafiles.- Row-Major vs. Column-Major Order.- Images as 2D Matrix Datafiles.- Advantages of 2D Matrix Datafiles.- 2D Matrix Grids.- Uniform Grids and No Grids.- Non-Uniform Grids.- Warped Grids.- Sparse Grids.- The Dimensionality of 2D Matrix Data.- 2D Matrix Files as Column Files.- Column Files as 2D Matrix Files.- Visualizing 2D Matrix Data.- Color Raster Imaging.- Surface Plots.- Contour Plots.- Vector Plots.- Summary.- 9. 3D Matrix (Volumetric) Data.- 3D Matrix Datafiles on Disk.- ASCII Text and Binary 3D Matrix Datafiles.- 3D Matrix Grids.- The Dimensionality of 3D Matrix Data.- 3D Matrix File from a Series of 2D Matrix Files.- Series of 2D Matrix Files from a 3D Matrix File.- Visualizing 3D Matrix Data.- Slicing and Dicing.- Isosurfaces.- Volumetric Visualization.- Vector Fields and Streamlines.- Summary.- 10. Polygonal Data.- Unstructured Grids vs. Structured Grids.- Structured Grids.- Unstructured Grids.- Node-Based Data vs. Cell-Based Data.- Polygonal Data on Disk.- A Cube Example.- Binary vs. ASCII Text.- From Surfaces to Solids.- Visualizing Polygonal Data.- Polygonal to 3D Matrix.- Photorealistic Rendering.- Summary.- 11. Conversions and Adding Dimensions.- Column Data to 2D Matrix Data.- Step 1: Gridding Scatter Data.- A 3 by 3 Grid Example.- A 4 by 4 Grid Example.- Selecting Destination Grid Size.- Reducing Bin Averaging.- Reducing Missing Values.- Missing Data Rags.- Step 2: Filling Missing Data.- Nearest Neighbor Fill.- Linear Interpolation.- Linear Interpolation Along Rows.- Linear Interpolate Along Columns and Rows.- Smooth Fill.- Weighted Fill.- Weighted Factor Functions.- Cutoff Radius.- Secondary Passes.- Comparing Weighted Fill and Smooth Fill.- Weighted Fill: Pros and Cons.- Kriging.- Variances.- Creating the Variance Matrix.- Minimizing the Variance Matrix.- Co-Linear Points.- Subdividing the Region.- A Small Kriging Example.- A Larger Kriging Example.- Kriging Summary.- Other Conversions.- Column Data to 3D Matrix Data.- Polygonal Data to Matrix Data.- Matrix Data to Column Data.- 2D Matrix to 3D Matrix.- 3D Matrix to 2D Matrix.- Adding Dimensions.- 1D Data to 2D Data: An Example.- Summary.- IV Data Formats.- 12. Data Formats Overview.- Scientific Data and Graphics File Formats—Introduction.- The Importance of Standards.- Scientific Data Formats—The First Revolution.- Scientific Data Formats vs. Graphics Formats.- Graphics Formats.- Home Growing a Bitmap File Format.- The SILI Format.- Data Compression.- Home Growing a Vector File Format.- Binary vs. ASCII Text Encoded Vector Graphics Files.- Bitmap Graphics Formats vs. Vector Graphics Formats.- Scientific Data Formats.- Annotations.- Data Organization.- Levels of Abstraction.- Data Models.- Organizational Concepts.- Sequential Organization.- Rat File Organization.- Hierarchical Organization.- Relational Organization.- Object-Oriented Organization.- Evaluating Data Formats.- Type of Format.- Hardware Issues.- Features.- Availability and Support.- Summary.- 13. Popular Graphics File Formats.- CGM (Computer Graphics Metafile).- Summary.- Overview of Disk Format.- References.- DXF (Data Interchange Format).- Summary.- Overview of Disk Format.- References.- EPS (Encapsulated PostScript).- Summary.- Overview of Disk Format.- References.- GIF (Graphics Interchange Format).- Summary.- Overview of Disk Format.- References.- GKS (Graphical Kernel System).- Summary.- Overview of Subroutine Library.- References.- PHIGS (Programmer Hierarchical Interactive Graphics Sandard).- Summary.- Overview of Subroutine Library.- References.- PICT.- Summary.- Overview of Disk Format.- PICT Bitmaps.- References.- PostScript.- Summary.- Overview of Disk Format.- References.- TIFF (Tag Image File Format).- Summary.- Overview of Disk Format.- TIFF Tagged Pointers.- TIFF Image Strips.- TIFF Example.- References.- 14. Popular Scientific Data Formats.- BUFR (Binary Universal Format for Representation of Data).- GRIB (Gridded Binary Format).- Summary.- Overview of the BUFR Disk Format.- BUFR Data Descriptors.- Scaling of BUFR Data Values.- A BUFR Example.- Overview of the GRIB Disk Format.- GRIB Section 1—Product Definition.- GRIB Section 2—Grid Definition.- GRIB Section 3—Bitmap.- GRIB Section 4—Data.- References.- CDF (Common Data Format).- Summary.- Overview of the CDF Format.- CDF Records and Variables.- CDF Variances.- CDF Virtual Values.- CDF Attributes, Data Types, Encoding, and Datafiles.- CDF Subroutine Library.- CDF Skeleton Files.- References.- DLIS (Digital Log Interchange Standard).- RP66 (Recommended Practice 66).- Summary.- Overview of the DLIS Format.- DLIS Disk Files.- DLIS Logical Records.- DLIS Logical Record Segments.- DLIS Objects.- DLIS Components.- DLIS Summary.- References.- FITS (Flexible Image Transport System).- Summary.- Overview of Disk Format.- FITS Header.- FITS Records.- Optional FITS Keywords.- FITS Bintables.- References.- HDF (Hierarchical Data Format).- Summary.- Overview of the HDF Subroutine Library.- Overview of the HDF Disk Format.- HDF Directory Structure.- HDF Data Descriptors.- HDF Data Descriptor Tags.- HDF Data Descriptor Reference Numbers.- HDF Groups.- HDF Scientific Datasets.- HDF Vdata Records.- HDF Extended Tags.- References.- netCDF (Network Common Data Form).- Summary.- Overview of the netCDF Format.- CDF Dimensions and Dimensionality.- CDL (Network Common Data Form Language.- References.- PDS (Planetary Data System).- Summary.- Overview of the PDS Format.- PDS Database Organization.- PDS Datafile Organization.- References.- Plot3D.- Summary.- Overview of the Plot3D Format.- References.- SEG-Y (SEG Digital Tape Format Y).- SEG-A (SEG Digital Tape Format A).- SEG-B (SEG Digital Tape Format B).- SEG-C (SEG Digital Tape Format C).- SEG-D (SEG Digital Tape Format D).- Summary.- Overview of the SEG Formats.- Overview of the SEG-Y Format.- References.- 15. Additional Formats.- Candis (C-Language for Analysis and Display).- References.- DICOM (Digital Imaging and Communications in Medicine).- ACR-NEMA (Am. Coll. of Radiology—Nat. Elect. Manuf. Assoc.).- References.- Erdas.- Flux.- GF3 (General Format 3).- IGES (Initial Graphics Exchange Specification).- PDES (Product Data Exchange Using STEP).- PBM (Portable Bitmap).- PGM (Portable Gray Map).- PPM (Portable Pixel Map).- PNM (Portable Any Map).- PCX.- PDB.- SBF (SERI Broadband Format).- SDTS (Spatial Data Transfer Standard).- FIPS 173 (Federal Information Processing Standard 173).- References.- SNAP (Simplified Numerical Analysis Postprocessing).- WMF (Microsoft Windows Metafile).- XBM (X Window Bitmap).- XWD (X Window Dump).- V Bringing It All Together.- Bringing It All Together.- Judy ReSyrch—Fan Simulation.- Michael Astroe—FITS Data.- Dr. Tim Boans—MRI Scans.- Jeanne Beeker—Solubility Experiments.- Wolfram Herth—Ground Water Information.- Summary.