<html><head><META http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Chapter 6. Text Tables</title><link href="guide.css" rel="stylesheet" type="text/css"><meta content="DocBook XSL Stylesheets V1.65.1" name="generator"><meta name="keywords" content="Text, Tables"><meta name="keywords" content="Hsqldb, Hypersonic, Database, JDBC, Java"><link rel="home" href="index.html" title="Hsqldb User Guide"><link rel="up" href="index.html" title="Hsqldb User Guide"><link rel="previous" href="ch05.html" title="Chapter 5. Deployment Issues"><link rel="next" href="ch07.html" title="Chapter 7. TLS"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table summary="Navigation header" width="100%"><tr><th align="center" colspan="3">Chapter 6. Text Tables</th></tr><tr><td align="left" width="20%"><a accesskey="p" href="ch05.html"><img src="navicons/prev.gif" alt="Prev"></a> </td><th align="center" width="60%"> </th><td align="right" width="20%"> <a accesskey="n" href="ch07.html"><img src="navicons/next.gif" alt="Next"></a></td></tr></table><hr></div><div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="texttables-chapter"></a>Chapter 6. Text Tables</h2></div><div><h3 class="subtitle"><i>Text Tables as a Standard Feature of Hsqldb</i></h3></div><div><div class="authorgroup"><div class="author"><h3 class="author"><span class="firstname">Bob</span> <span class="surname">Preston</span></h3><div class="affiliation"><span class="orgname">HSQLDB Development Group<br></span></div></div><div class="author"><h3 class="author"><span class="firstname">Fred</span> <span class="surname">Toussi</span></h3><div class="affiliation"><span class="orgname">HSQLDB Development Group<br></span></div><tt class="email"><<a href="mailto:ft@cluedup.com">ft@cluedup.com</a>></tt></div></div></div><div><div class="legalnotice"><p>Copyright 2002-2005 Bob Preston and Fred Toussi. Permission is granted to distribute this document without any alteration under the terms of the HSQLDB license. Additional permission is granted to the HSQLDB Development Group to distribute this document with or without alterations under the terms of the HSQLDB license.</p></div></div><div><p class="pubdate">$Date: 2007/08/28 13:19:09 $</p></div></div><div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="ch06.html#N10F46">The Implementation</a></span></dt><dd><dl><dt><span class="section"><a href="ch06.html#N10F49">Definition of Tables</a></span></dt><dt><span class="section"><a href="ch06.html#N10F58">Scope and Reassignment</a></span></dt><dt><span class="section"><a href="ch06.html#N10F73">Null Values in Columns of Text Tables</a></span></dt><dt><span class="section"><a href="ch06.html#N10F7F">Configuration</a></span></dt><dt><span class="section"><a href="ch06.html#disconnecting_text_tables">Disconnecting Text Tables</a></span></dt></dl></dd><dt><span class="section"><a href="ch06.html#N1104D">Text File Issues</a></span></dt><dt><span class="section"><a href="ch06.html#N11075">Text File Global Properties</a></span></dt><dt><span class="section"><a href="ch06.html#N110B3">Importing from a Text Table file</a></span></dt></dl></div><p>Text Table support for HSQLDB was originally developed by Bob Preston independently from the Project. Subsequently Bob joined the Project and incorporated this feature into version 1.7.0, with a number of enhancements, especially the use of conventional SQL commands for specifying the files used for Text Tables.</p><p>In a nutshell, Text Tables are CSV or other delimited files treated as SQL tables. Any ordinary CSV or other delimited file can be used. The full range of SQL queries can be performed on these files, including SELECT, INSERT, UPDATE and DELETE. Indexes and unique constraints can be set up, and foreign key constraints can be used to enforce referential integrity between Text Tables themselves or with conventional tables.</p><p>HSQLDB with Text Table support is the only comprehensive solution that employs the power of SQL and the universal reach of JDBC to handle data stored in text files and will have wide-ranging use way beyond the currently established Java realm of HSQLDB.</p><div class="orderedlist"><p class="title"><b>Goals of the Implementation</b></p><ol type="1"><li><p>We aimed to finalise the DDL for Text Tables so that future releases of HSQLDB use the same DDL scripts.</p></li><li><p>We aimed to support Text Tables as GLOBAL TEMPORARY or GLOBAL BASE tables in the SQL domain.</p></li></ol></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="N10F46"></a>The Implementation</h2></div></div><div></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="N10F49"></a>Definition of Tables</h3></div></div><div></div></div><p>Text Tables are defined similarly to conventional tables with the added TEXT keyword:</p><pre class="programlisting"> CREATE TEXT TABLE <tablename> (<column definition> [<constraint definition>])</pre><p>In addition, a SET command specifies the file and the separator character that the Text table uses:</p><pre class="programlisting"> SET TABLE <tablename> SOURCE <quoted_filename_and_options> [DESC]</pre><p>Text Tables cannot be created in memory-only databases (databases that have no script file).</p></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="N10F58"></a>Scope and Reassignment</h3></div></div><div></div></div><div class="itemizedlist"><ul type="disc"><li><p>A Text table without a file assigned to it is READ ONLY and EMPTY.</p></li><li><p>A Temporary Text table has the scope and the lifetime of the SQL session (a JDBC Connection).</p></li><li><p>Reassigning a Text Table definition to a new file has implications in the following areas:</p><div class="orderedlist"><ol type="1"><li><p>The user is required to be an administrator.</p></li><li><p>Existing transactions are committed at this point.</p></li><li><p>Constraints, including foreign keys referencing this table, are kept intact. It is the responsibility of the administrator to ensure their integrity.</p></li></ol></div><p>From version 1.7.2 the new source file is scanned and indexes are built when it is assigned to the table. At this point any violation of NOT NULL, UNIQUE or PRIMARY KEY constrainst are caught and the assignment is aborted. However, foreign key constraints are not checked at the time of assignment or reassignment of the source file.</p></li></ul></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="N10F73"></a>Null Values in Columns of Text Tables</h3></div></div><div></div></div><p>This has changed since 1.7.2 to support both null values and empty strings.</p><div class="itemizedlist"><ul type="disc"><li><p>Empty fields are treated as NULL. These are fields where there is nothing or just spaces between the separators.</p></li><li><p>Quoted empty strings are treated as empty strings.</p></li></ul></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="N10F7F"></a>Configuration</h3></div></div><div></div></div><p>The default field separator is a comma (,). A different field separator can be specified within the SET TABLE SOURCE statement. For example, to change the field separator for the table mytable to a vertical bar, place the following in the SET TABLE SOURCE statement, for example:</p><div class="informalexample"><pre class="programlisting"> SET TABLE mytable SOURCE "myfile;fs=|"</pre></div><p>Since HSQLDB treats CHAR's, VARCHARs, and LONGVARCHARs the same, the ability to assign different separators to the latter two is provided. When a different separator is assigned to a VARCHAR or LONGVARCHAR field, it will terminate any CSV field of that type. For example, if the first field is CHAR, and the second field LONGVARCHAR, and the separator fs has been defined as the pipe (|) and vs as the period (.) then the data in the CSV file for a row will look like:</p><pre class="screen"> First field data|Second field data.Third field data</pre><p>The following example shows how to change the default separator to the pipe (|), VARCHAR separator to the period (.) and the LONGVARCHAR separator to the tilde (~). Place the following within the SET TABLE SOURCE statement, for example:</p><div class="informalexample"><pre class="programlisting"> SET TABLE mytable SOURCE "myfile;fs=|;vs=.;lvs=~"</pre></div><p>HSQLDB also recognises the following special indicators for separators:</p><div class="variablelist"><p class="title"><b>special indicators for separators</b></p><dl><dt><span class="term">\semi</span></dt><dd><p>semicolon</p></dd><dt><span class="term">\quote</span></dt><dd><p>qoute</p></dd><dt><span class="term">\space</span></dt><dd><p>space character</p></dd><dt><span class="term">\apos</span></dt><dd><p>apostrophe</p></dd><dt><span class="term">\n</span></dt><dd><p>newline - Used as an end anchor (like $ in regular expressions)</p></dd><dt><span class="term">\r</span></dt><dd><p>carriage return</p></dd><dt><span class="term">\t</span></dt><dd><p>tab</p></dd><dt><span class="term">\\</span></dt><dd><p>backslash</p></dd><dt><span class="term">\u####</span></dt><dd><p>a Unicode character specified in hexadecimal</p></dd></dl></div><p>Furthermore, HSQLDB provides csv file support with three additional boolean options: <tt class="literal">ignore_first</tt>, <tt class="literal">quoted</tt> and <tt class="literal">all_quoted</tt>. The <tt class="literal">ignore_first</tt> option (default false) tells HSQLDB to ignore the first line in a file. This option is used when the first line of the file contains column headings. The <tt class="literal">all_quoted</tt> option (default false) tells the program that it should use quotes around all character fields when writing to the source file. The <tt class="literal">quoted</tt> option (default true) uses quotes only when necessary to distinguish a field that contains the separator character. It can be set to false to prevent the use of quoting altogether and treat quote characters as normal characters. These options may be specified within the <tt class="literal">SET TABLE SOURCE</tt> statement:</p><pre class="programlisting"> SET TABLE mytable SOURCE "myfile;ignore_first=true;all_quoted=true"</pre><p>When the default options <tt class="literal">all_quoted=</tt> <tt class="literal">false</tt> and <tt class="literal">quoted=true</tt> are in force, fields that are written to a line of the csv file will be quoted only if they contain the separator or the quote character. The quote character is doubled when used inside a string. When <tt class="literal">all_quoted=false</tt> and <tt class="literal">quoted=false</tt> the quote character is not doubled. With this option, it is not possible to insert any string containing the separator into the table, as it would become impossible to distinguish from a separator. While reading an existing data source file, the program treats each individual field separately. It determines that a field is quoted only if the first character is the quote character. It interprets the rest of the field on this basis.</p><p>The character encoding for the source file is<tt class="literal"> ASCII </tt>by default. To support UNICODE or source files preprared with different encodings this can be changed to <tt class="literal">UTF-8</tt> or any other encoding. The default is <tt class="literal">encoding=ASCII </tt>and the option <tt class="literal">encoding=UTF-8</tt> or other supported encodings can be used.</p><p>Finally, HSQLDB provides the ability to read a text file from the bottom up and making them READ ONLY, by placing the keyword "DESC" at the end of the SET TABLE SOURCE statement:</p><pre class="programlisting"> SET TABLE mytable SOURCE "myfile" DESC</pre><p>This feature provides functionality similar to the Unix tail command, by re-reading the file each time a select is executed. Using this feature sets the table to read-only mode. Afterwards, it will no longer be possible to change the read-only status with <tt class="literal">SET TABLE <tablename> READONLY TRUE</tt>.</p><p>Text table source files are cached in memory. The maximum number of rows of data that are in memory at any time is controlled by the <tt class="literal">textdb.cache_scale </tt> property. The default value for <tt class="literal">textdb.cache_scale</tt> is 10 and can be changed by setting the property in the .properties file for the database. The number of rows in memory is calculated as 3*(2**scale), which translates to 3072 rows for the default textdb.cache_scale setting (10). The property can also be set for individual text tables:</p><pre class="programlisting"> SET TABLE mytable SOURCE "myfile;ignore_first=true;all_quoted=true;cache_scale=12"</pre></div><div class="section" lang="en"><div class="titlepage"><div><div><h3 class="title"><a name="disconnecting_text_tables"></a>Disconnecting Text Tables</h3></div></div><div></div></div><p>The following describes behaviour present in 1.8.0.8 and later.</p><p>Text tables may be <span class="emphasis"><em>disconnected</em></span> from their underlying data source, i.e. the text file.</p><p>You can explicitly disconnect a text table from its file by issuing the following statement: <pre class="programlisting"> SET TABLE mytable SOURCE OFF</pre> </p><p>Subsequently, <tt class="literal">mytable</tt> will be empty and read-only. However, the data source description will be preserved, and the table can be re-connected to it with <pre class="programlisting"> SET TABLE mytable SOURCE ON</pre> </p><p>When a database is opened, if the source file for an existing text table is missing the table remains disconnected from its data source, but the source sescription is preserved. This allows the missing source file to be added to the directory and the table re-connected to it with the above command. </p></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="N1104D"></a>Text File Issues</h2></div></div><div></div></div><div class="itemizedlist"><p class="title"><b>Text File Issues</b></p><ul type="disc"><li><p>File locations are restricted to below the directory that contains the database, unless the textdb.allow_full_path property is set true in the database properties file.</p></li><li><p>Blank lines are allowed anywhere in the text file, and are ignored.</p></li><li><p>The file location for a text table created with</p><pre class="programlisting"> SELECT <select list> INTO TEXT <tablename> FROM</pre><p>is the directory that contains the database and the file name is based on the table name. The table name is converted into the file name by replacing all the non-alphanumeric characters with the underscore character, conversion into lowercase, and adding the ".csv" suffix.</p></li><li><p>It is possible to define a primay key or identity column for text tables.</p></li><li><p>When a table source file is used with the<tt class="literal"> ignore_first=true </tt>option, the first, ignored line is replaced with a blank line after a SHUTDOWN COMPACT.</p></li><li><p>An existing table source file may include CHARACTER fields that do not begin with the quote character but contain instances of the quote character. These fields are read as literal strings. Alternatively, if any field begins with the quote character, then it is interpreted as a quoted string that should end with the quote character and any instances of the quote character within the string is doubled. When any field containing the quote character or the separator is written out to the source file by the program, the field is enclosed in quote character and any instance of the quote character inside the field is doubled.</p></li><li><p>Inserts or updates of CHARACTER type field values are allowed with strings that contains the linefeed or the carriage return character. This feature is disabled when both quoted and all_quoted properties are false.</p></li><li><p>ALTER TABLE commands that add or drop columns are not supported with non-empty text tables.</p></li></ul></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="N11075"></a>Text File Global Properties</h2></div></div><div></div></div><div class="itemizedlist"><p class="title"><b>Complete list of supported global properties in *.properties files</b></p><ul type="disc"><li><p> <tt class="literal">textdb.fs</tt> </p></li><li><p> <tt class="literal">textdb.lvs</tt> </p></li><li><p> <tt class="literal">textdb.quoted</tt> </p></li><li><p> <tt class="literal">textdb.all_quoted</tt> </p></li><li><p> <tt class="literal">textdb.ignore_first</tt> </p></li><li><p> <tt class="literal">textdb.encoding</tt> </p></li><li><p> <tt class="literal">textdb.cache_scale</tt> </p></li><li><p> <tt class="literal">textdb.allow_full_path</tt> </p></li></ul></div></div><div class="section" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="N110B3"></a>Importing a Text Table file in to a Traditional (non-Text Table) Table</h2></div></div><div></div></div><p>The directory <tt class="filename">src/org/hsqldb/sample</tt> in your HSQLDB distibution contains a file named <tt class="filename">load_binding_lu.sql</tt>. This is a working SQL file which imports a pipe-delimited text file from the database's file directory into an existing normal table. You can edit a copy of this file and use it directly with <a href="ch08.html" title="Chapter 8. SqlTool">SqlTool</a>, or you can use the SQL therein as a model (using any SQL client at all).</p></div></div><div class="navfooter"><hr><table summary="Navigation footer" width="100%"><tr><td align="left" width="40%"><a accesskey="p" href="ch05.html"><img src="navicons/prev.gif" alt="Prev"></a> </td><td align="center" width="20%"><a accesskey="u" href="index.html"><img src="navicons/up.gif" alt="Up"></a></td><td align="right" width="40%"> <a accesskey="n" href="ch07.html"><img src="navicons/next.gif" alt="Next"></a></td></tr><tr><td valign="top" align="left" width="40%">Chapter 5. Deployment Issues </td><td align="center" width="20%"><a accesskey="h" href="index.html"><img src="navicons/home.gif" alt="Home"></a></td><td valign="top" align="right" width="40%"> Chapter 7. TLS</td></tr></table></div></body></html>