Changed Data Model

------------------

* Created the extended data table. This is a more generic version of the
  extended data field that was on ``ts_user``. Instead of arbitrary key-value
  pairs going in a list on the user record we will have an additional table, a
  one to many relationship between existing tables and the ``ts_ext_data``
  table, each row being an extended value of the corresponding record.
* Added support to the ``timestamper:create_tables/1`` and
  ``timestamper_dev:create_table/1`` functions for the new ``ts_ext_data`` table.

Documentation
-------------

* Added some general docs about the DB layer code.
* Added two new issues, *D0020*: Entry exclusion filters and *D0021*: notes width.

API Changes
-----------

Necessitated by the change to the data model.

* Updated ``ts_api`` data-retrieval functions to use extended data:

    * ``get_user_summary/2``
    * ``list_timelines/2``
    * ``list_entries/3``

* Updated ``ts_api:put_timeline/3`` to parse the extended data supplied by the
  caller. *FIXME* it is not actually saving this data.
* TODO: Started updating ``ts_api:post_entry/3`` to handle extended data, need
  to finish.

Database Layer
--------------

Changes necessitated by the change to the model.

* Added ``ts_common:do_set_ext_data/2`` which iterates through the extended data
  key-value pairs calling ``ts_ext_data:set_property/3``. It does not provide a
  transaction context.
* Split ``ts_common:new/1`` and ``ts_common:update/1`` functions into multiple
  functions to prevent code-duplication:

    * ``do_{new,update}`` contains the code that performs integrity checks and
      the actual database write function. It does this assuming that it is being
      called from within the context of an mnesia transaction (uses
      ``mnesia:read`` and ``mnesia:write``).
    * ``{new,update}/1`` performs the same function as previously. The
      implementation changed from using mnesia ``dirty_*`` calls to prociding a
      transaction and calling ``do_{new,update}/1``.
    * ``{new,update}/2`` expect the record to update/create and the extended
      data to write atomically with the record. They provide a transaction
      context, call ``do_{new,update}/1``, then call ``do_set_ext_data/2``.

* Similar to the refactoring of ``ts_common:{new,update}/1``,
  ``ts_entry:{new,update}/1`` have been refactored into multiple methods each to
  support extended data properties:

    * ``do_{new,update}/1`` perform the actual update assuming we have already
      established an mnesia transaction.
    * ``{new,update}/1`` behave the same as they used to, but now do so by
      creating an mnesia transaction and calling ``do_{new,update}/1``.
    * ``{new,update}/2`` create an mnesia transaction, call
      ``do_{new,update}/1``, and then call ``ts_common:do_set_ext_data/2``.

* Again similar to the refactoring of ``ts_common:{new,update}/1``,
  ``ts_user:{new,update}/1`` have been refactored into multiple methods each to
  support extended data properties:

    * ``do_{new,update}/1`` perform the actual update assuming we have already
      established an mnesia transaction.
    * ``{new,update}/1`` behave the same as they used to, but now do so by
      creating an mnesia transaction and calling ``do_{new,update}/1``.
    * ``{new,update}/2`` create an mnesia transaction, call
      ``do_{new,update}/1``, and then call ``ts_common:do_set_ext_data/2``.

* Created the ``ts_ext_data`` module as the interface to the extended data
  properties introduced in the data model:

    * ``create_table/1`` performs the same function as it does in the other db
      layer modules, creates the table with the appropriate structure given the
      more general table options desired (location, storage type, etc.).
    * ``set_property/3`` takes a record, a property key, and a property value as
      input and sets the property described by the property key on the record to
      the given value, assuming this is a valid property for the record to have.
      For example, currently the ``ts_user`` record can have an associated
      property, ``last_timeline``, which represents the last timeline the user
      was working with. Trying to pass this property with a ``ts_entry`` record
      would result in an exception. This function uses
      ``ts_ext_data:do_set_property/3`` as its underlying implementation.
    * ``get_property/2`` takes a record and a property key and returns the value
      of that property for the given record, or ``not_set`` if the property has
      not been set on that record. This method creates its own mnesia
      transaction.
    * ``get_properties/1`` takes a record and returns a list of key-value tuples
      representing all of the extended data properties set for the given record.
      This method creates its own mnesia transaction.
    * ``do_set_property/3`` takes a record reference (not the whole record), a
      a property key, and adds the property assignment to the ``ts_ext_data``
      table. It creates its own mnesia transaction.

* Added ``new/2`` and ``update/2`` to the ``ts_timeline`` module to support
  extended data properties. They delegate implementation to
  ``ts_common:{new,update}/2``.

JSON Encoding/Decoding
----------------------

Changes necessitated by the change to the data model.

The JSON objects now contain a potentially unlimited number of fields, as each
extended data property is encoded as a seperate field, and looks no different
from any of the required fields on the object. The intended explanation in API
documentation is that each object type (``user``, ``timeline``, or ``entry``)
now has both *required* fields that *MUST* be present in every message in either
direction and *optional* fields that may or may not be present in any
communication with the API. There should be a clear distinction between which
fields are required and which are optional. It might also be a good idea to
provide a suggested default for optional values when they are not present.

* Updated documentation about JSON record structures to reflect the fact that
  there are now potentially many optional attributes in addition to the required
  attributes for each record.
* ``record_to_ejson/1`` refactored to ``record_to_ejson/2`` which also takes
  the extended data attributes and appends them as additional attributes to
  the end of the record structure.
* Created ``ext_data_to_ejson/{1,2}`` to provide a mechanism for reformatting
  extended data properties whose internal representations are not immediately
  translatable into JSON. Currently only the ``entry_exclusions`` property,
  which is a list of strings, needs to be treated this way (changing from
  ``[val, val]`` to ``{array, [val, val]}`` as needed by ``json:encode/1``.
  ``ext_data_to_ejson/1`` acts as a more user-friendly facade to
  ``ext_data_to_ejson/2``.
* Rewrote ``ejson_to_record/{2,3}`` and ``ejson_to_record_strict/{2,3}`` to
  handle extended data. They now use a common method, ``construct_record/3`` to
  create the actual record object and extended data key-value list.
  ``ejson_to_record_strict/{2,3}`` only differs in that it checks for the
  presence of each required field of the record after the record is constructed.
  The three-parameter versions of these functions also take in the intended
  reference for the constructed record, replacing anything that is in the EJSON
  body as the record reference (useful when the body does not have the record
  ids). These methods now return a tuple: ``{Record, ExtData}`` instead of just
  the record.
* Created ``construct_record/3`` takes a record and the EJSON fields from the
  input object. The third parameter is an accumulator for the extended data
  properties found when constructing the record. This method works by iterating
  over the list of input fields. It recognizes any required fields and updates
  the record being built with the value. Any fields it does not recognize it
  assumes are extended data properties and adds to its list. When all input
  fields have been visited it returns the record and list it has constructed.
* ``ejson_to_ext_data/{1,2}`` is the inverse of ``ext_data_to_ejson/{1,2}``.
  *TODO*: this method is not actually being used by the ``ejson_to_record*``
  methods.

.ide/vim-views/ts_common.erl.view

@@ -0,0 +1,107 @@
+let s:so_save = &so | let s:siso_save = &siso | set so=0 siso=0
+argglobal
+edit ~/projects/jdb-labs/timestamper/web-app/src/ts_common.erl
+setlocal keymap=
+setlocal noarabic
+setlocal autoindent
+setlocal balloonexpr=
+setlocal nobinary
+setlocal bufhidden=
+setlocal buflisted
+setlocal buftype=
+setlocal nocindent
+setlocal cinkeys=0{,0},0),:,0#,!^F,o,O,e
+setlocal cinoptions=
+setlocal cinwords=if,else,while,do,for,switch
+setlocal comments=s1:/*,mb:*,ex:*/,://,b:#,:%,:XCOMM,n:>,fb:-
+setlocal commentstring=/*%s*/
+setlocal complete=.,w,b,u,t,i
+setlocal completefunc=
+setlocal nocopyindent
+setlocal nocursorcolumn
+setlocal nocursorline
+setlocal define=
+setlocal dictionary=
+setlocal nodiff
+setlocal equalprg=
+setlocal errorformat=
+setlocal expandtab
+if &filetype != 'erlang'
+setlocal filetype=erlang
+endif
+setlocal foldcolumn=0
+setlocal foldenable
+setlocal foldexpr=0
+setlocal foldignore=#
+setlocal foldlevel=0
+setlocal foldmarker={{{,}}}
+setlocal foldmethod=manual
+setlocal foldminlines=1
+setlocal foldnestmax=20
+setlocal foldtext=foldtext()
+setlocal formatexpr=
+setlocal formatoptions=tcq
+setlocal formatlistpat=^\\s*\\d\\+[\\]:.)}\\t\ ]\\s*
+setlocal grepprg=
+setlocal iminsert=2
+setlocal imsearch=2
+setlocal include=
+setlocal includeexpr=
+setlocal indentexpr=
+setlocal indentkeys=0{,0},:,0#,!^F,o,O,e
+setlocal noinfercase
+setlocal iskeyword=@,48-57,_,192-255
+setlocal keywordprg=
+setlocal nolinebreak
+setlocal nolisp
+setlocal nolist
+setlocal makeprg=
+setlocal matchpairs=(:),{:},[:]
+setlocal nomodeline
+setlocal modifiable
+setlocal nrformats=octal,hex
+setlocal number
+setlocal numberwidth=4
+setlocal omnifunc=
+setlocal path=
+setlocal nopreserveindent
+setlocal nopreviewwindow
+setlocal quoteescape=\\
+setlocal noreadonly
+setlocal norightleft
+setlocal rightleftcmd=search
+setlocal noscrollbind
+setlocal shiftwidth=4
+setlocal noshortname
+setlocal nosmartindent
+setlocal softtabstop=0
+setlocal nospell
+setlocal spellcapcheck=[.?!]\\_[\\])'\"\	\ ]\\+
+setlocal spellfile=
+setlocal spelllang=en
+setlocal statusline=
+setlocal suffixesadd=
+setlocal swapfile
+setlocal synmaxcol=3000
+if &syntax != 'erlang'
+setlocal syntax=erlang
+endif
+setlocal tabstop=4
+setlocal tags=
+setlocal textwidth=80
+setlocal thesaurus=
+setlocal nowinfixheight
+setlocal nowinfixwidth
+setlocal wrap
+setlocal wrapmargin=0
+silent! normal! zE
+let s:l = 16 - ((14 * winheight(0) + 23) / 46)
+if s:l < 1 | let s:l = 1 | endif
+exe s:l
+normal! zt
+16
+normal! 034l
+let &so = s:so_save | let &siso = s:siso_save
+doautoall SessionLoadPost
+" vim: set ft=vim :
+syntax on

doc/db_layer.rst

@@ -0,0 +1,24 @@
+TimeStamper DB Layer
+====================
+
+The following modules make up the database layer:
+
+* ``ts_user``: Interface to user data.
+* ``ts_timeline``: Interface to timeline data.
+* ``ts_entry``: Interface to timeline entry data.
+* ``ts_ext_data``: Interface to extended data that can be set on different
+  records.
+* ``ts_db_records``: Definition of data records.
+
+The following modules and files are implementation details of the DB layer:
+
+* ``id_counter``: Adds support for unique, sequential ID generation.
+* ``ts_common``: Provides the implementation for any operations that are common
+  between the different interfaces.
+
+Philosophy
+----------
+
+The database layer should abstract all database-specific code away from the
+caller. Users of the DB layer should not have to think about transactions,
+locking, etc.

doc/issues/desktop/0020fn5.rst

@@ -0,0 +1,10 @@
+Add exclusion filter for entries.
+=================================
+
+Add a way for users to add regexes for entries to be ignored in the
+display. Exclusions may be per-timeline or per-user.
+
+========= ==========
+Created:  2011-06-10
+Resolved: YYYY-MM-DD
+========= ==========

doc/issues/desktop/0021tn5.rst

@@ -0,0 +1,7 @@
+Constrain width of notes fields to width of mark.
+=================================================
+
+========= ==========
+Created:  2011-06-10
+Resolved: YYYY-MM-DD
+====================

src/timestamper.erl

@@ -13,4 +13,5 @@ create_tables(Nodes) ->
     {atomic, ok} = ts_user:create_table(TableOpts),
     {atomic, ok} = ts_timeline:create_table(TableOpts),
     {atomic, ok} = ts_entry:create_table(TableOpts),
+    {atomic, ok} = ts_ext_data:create_table(TableOpts),
     ok.

src/timestamper_dev.erl

@@ -14,4 +14,5 @@ create_tables(Nodes) ->
     {atomic, ok} = ts_user:create_table(TableOpts),
     {atomic, ok} = ts_timeline:create_table(TableOpts),
     {atomic, ok} = ts_entry:create_table(TableOpts),
+    {atomic, ok} = ts_ext_data:create_table(TableOpts),
     ok.

src/ts_api.erl

@@ -204,11 +204,17 @@ get_user_summary(YArg, Username) ->
     case ts_user:lookup(Username) of
         no_record -> make_json_404(YArg);
         User ->
-            EJSONUser = ts_json:record_to_ejson(User),
+            UserExtData - ts_ext_data:get_properties(Username),
+            EJSONUser = ts_json:record_to_ejson(User, UserExtData),
 
             Timelines = ts_timeline:list(Username, 0, 100),
             EJSONTimelines = {array,
-                lists:map(fun ts_json:record_to_ejson/1, Timelines)},
+                lists:map(
+                    fun(Timeline) ->
+                        ts_json:record_to_ejson(Timeline,
+                            ts_ext_data:get_properties(Timeline#ts_timeline.ref))
+                    end,
+                    Timelines)},
 
             JSONResp = json:encode({struct,
                [{user, EJSONUser},
@@ -245,7 +251,12 @@ list_timelines(YArg, Username) ->
     Timelines = ts_timeline:list(Username, Start, Length),
 
     % convert them all to their EJSON form
-    EJSONTimelines = {array, lists:map(fun ts_json:record_to_ejson/1, Timelines)},
+    EJSONTimelines = {array, lists:map(
+        fun (Timeline) ->
+            ts_json:record_to_ejson(Timeline, 
+                ts_ext_data:get_properties(Timeline#ts_timeline.ref))
+        end,
+        Timelines)},
 
     % create resposne
     JSONResponse = json:encode(EJSONTimelines),
@@ -268,12 +279,13 @@ put_timeline(YArg, Username, TimelineId) ->
     %{struct, Fields} = EJSON,
 
     % parse into a timeline record
-    TR = try ts_json:ejson_to_record_strict(
+    {TR, ExtData} =
+        try ts_json:ejson_to_record_strict(
             #ts_timeline{ref={Username, TimelineId}}, EJSON)
-         catch _:InputError ->
+        catch throw:{InputError, _StackTrace} ->
             error_logger:error_report("Bad input: ~p", [InputError]),
-            throw(make_json_400(YArg))
+            throw(make_json_400(YArg, {request_error, InputError))
-         end,
+        end,
 
     ts_timeline:write(TR),
     make_json_200(YArg, TR).
@@ -321,7 +333,11 @@ list_entries(YArg, Username, TimelineId) ->
             end,
 
             EJSONEntries = {array, lists:map(
-                fun ts_json:record_to_ejson/1, Entries)},
+                fun (Entry) ->
+                    ts_json:record_to_ejson(Entry,
+                        ts_ext_data:get_properties(Entry#ts_entry.ref)),
+                end,
+                Entries)},
 
             JSONResponse = json:encode(EJSONEntries),
 
@@ -353,7 +369,11 @@ list_entries(YArg, Username, TimelineId) ->
             end,
 
             EJSONEntries = {array, lists:map(
-                fun ts_json:record_to_ejson/1, Entries)},
+                fun (Entry) ->
+                    ts_json:record_to_ejson(Entry,
+                        ts_ext_data:get_properties(Entry#ts_entry.ref)),
+                end,
+                Entries)},
 
             JSONResponse = json:encode(EJSONEntries),
 
@@ -374,16 +394,18 @@ post_entry(YArg, Username, TimelineId) ->
     EJSON = parse_json_body(YArg),
 
     % parse into ts_entry record
-    ER = try ts_json:ejson_to_record_strict(
+    {ER, ExtData} = try ts_json:ejson_to_record_strict(
             #ts_entry{ref = {Username, TimelineId, undefined}}, EJSON)
          catch _:InputError ->
             error_logger:error_report("Bad input: ~p", [InputError]),
             throw(make_json_400(YArg))
          end,
 
+    %% TODO; should entries and their properties be created atomically?
     case ts_entry:new(ER) of
         % record created
         {ok, CreatedRecord} ->
+            
             [{status, 201}, make_json_200(YArg, CreatedRecord)];
 
         % will not create, record exists
@@ -445,7 +467,8 @@ parse_json_body(YArg) ->
 
 %% Create a JSON 200 response.
 make_json_200(_YArg, Record) ->
-    JSONResponse = json:encode(ts_json:record_to_ejson(Record)),
+    RecordExtData = ts_ext_data:get_properties(element(2, Record)),
+    JSONResponse = json:encode(ts_json:record_to_ejson(Record, RecordExtData)),
     {content, "application/json", JSONResponse}.
 
 make_json_400(YArg) -> make_json_400(YArg, []).

src/ts_common.erl

@@ -1,32 +1,64 @@
 -module(ts_common).
--export([new/1, update/1, update_record/2, list/3, order_datetimes/2]).
+-export([new/1, new/2, update/1, update/2, update_record/2, do_set_ext_data/2,
+         list/3, order_datetimes/2]).
 
 -include_lib("stdlib/include/qlc.hrl").
 
 new(Record) ->
+    {atomic, Result} = mnesia:transaction(fun() -> do_new(Record) end),
+    Result.
 
+new(Record, ExtData) when is_list(ExtData) ->
+    {atomic, Result} = mnesia:transaction(fun() ->
+        case do_new(Record) of
+            {error, Err} -> mnesia:abort({"Cannot create new record.", Err});
+            NewRecord -> case do_set_ext_data(Record, ExtData) of
+                    ok -> NewRecord;
+                    Error -> mnesia:abort(Error)
+            end
+        end
+    end),
+    Result.
+
+% required to be wrapped in a transaction
+do_new(Record) ->
     Table = element(1, Record),
 
     % check for existing record
-    case mnesia:dirty_read(Table, element(2, Record)) of
+    case mnesia:read(Table, element(2, Record)) of
         % record exists
         [ExistingRecord] -> {error, {record_exists, ExistingRecord}};
         
-        [] -> mnesia:dirty_write(Record)
+        [] -> mnesia:write(Record)
     end.
 
 update(Record) ->
+    {atomic, Result} = mnesia:transaction(fun() -> do_update(Record) end),
+    Result.
 
+update(Record, ExtData) when is_list(ExtData) ->
+    {atomic, Result} = mnesia:transaction(fun() ->
+        case do_update(Record) of
+            {error, Err} -> mnesia:abort({"Cannot update record.", Err});
+            UpdatedRecord -> case do_set_ext_data(Record, ExtData) of
+                of -> UpdatedRecord;
+                Error -> mnesia:abort({"Cannot update record.", Error})
+            end
+        end
+    end),
+    Result.
+
+do_update(Record) ->
     Table = element(1, Record),
     Key = element(2, Record),
 
     % look for existing record
-    case mnesia:dirty_read(Table, Key) of
+    case mnesia:read(Table, Key) of
         % record does not exist, cannot update
         [] -> no_record;
         % record does exist, update
         [ExistingRecord] ->
-            mnesia:dirty_write(update_record(ExistingRecord, Record))
+            mnesia:write(update_record(ExistingRecord, Record))
     end.
 
 update_record(Record, UpdateData) ->
@@ -66,6 +98,16 @@ list(Query, Start, Length) ->
     end),
     Result.
 
+% should be wrapped in a transaction
+do_set_ext_data(Record, ExtData) when is_list(ExtData) ->
+    Ref = element(2, Record),
+    lists:foreach(
+        fun({Key, Val}) ->
+            {atomic, ok} = ts_ext_data:set_property(Ref, Key, Val)
+        end,
+        ExtData)
+    ok.
+
 % This is somewhat ridiculous.
 order_datetimes({{Y1, Mon1, D1}, {H1, Min1, S1}},
                 {{Y2, Mon2, D2}, {H2, Min2, S2}}) ->

src/ts_db_records.hrl

@@ -4,12 +4,21 @@
     pwd_salt,
     name,
     email,
-    join_date,
+    join_date
-    ext_data = []    % other extensible data
 }).
 
-% ts_user.ext_data can be:
+% ts_user.ext_data format is
-%  [{last_timeline, TimelineId}]
+%  [{key, val}, {key, val}]
+%
+% Valid key/value pairs are:
+% 
+% * last_timeline/TimelineId
+%   
+%   list() representing a timeline owned by this user
+%
+% * entry_exclusions/EntryExclusions
+% 
+%   list() of regular expressions to be excluded from display, seperated by '|'
 
 -record(ts_timeline, {
     ref,    % {username, timelineid}
@@ -29,6 +38,11 @@
     expires
 }).
 
+% extensible data for arbitrary entities
+-record(ts_ext_data, {
+    ref,    % {ref, key}: reference with item ref and property key
+    value   % value
+}).
 %-record(ts_session, {
     %session_id,
     %expires,

src/ts_entry.erl

@@ -1,5 +1,6 @@
 -module(ts_entry).
--export([create_table/1, new/1, update/1, write/1, delete/1, lookup/3, list_asc/3, list_desc/3]).
+-export([create_table/1, new/1, new/2, update/1, update/2, write/1, delete/1,
+         lookup/3, list_asc/3, list_desc/3]).
 
 -include("ts_db_records.hrl").
 -include_lib("stdlib/include/qlc.hrl").
@@ -10,16 +11,28 @@ create_table(TableOpts) ->
                       {type, ordered_set}, {index, [timestamp]}]).
 
 new(ER = #ts_entry{}) ->
+    {atomic, NewRow} = mnesia:transaction(fun() -> do_new(ER) end),
+    {ok, NewRow}.
+
+new(ER = #ts_entry{}, ExtData) when is_list(ExtData) ->
     {atomic, NewRow} = mnesia:transaction(fun() ->
+        NewRow = do_new(ER),
+        ts_common:do_set_ext_data(ER, ExtData),
+        NewRow end),
+     NewRow.
+
+do_new(ER = #ts_entry{}) ->
         {Username, TimelineId, _} = ER#ts_entry.ref,
         NextId = id_counter:next_counter(ts_entry_id),
         NewRow = ER#ts_entry{ref = {Username, TimelineId, NextId}},
         ok = mnesia:write(NewRow),
-        NewRow end),
+        NewRow.
-    {ok, NewRow}.
 
 update(ER = #ts_entry{}) -> ts_common:update(ER).
 
+update(ER = #ts_entry{}, ExtData) when is_list(ExtData) ->
+    ts_common:update(ER, ExtData).
+    
 write(ER = #ts_entry{}) -> mnesia:dirty_write(ER).
 
 lookup(Username, TimelineId, EntryId) ->
@@ -30,7 +43,6 @@ lookup(Username, TimelineId, EntryId) ->
 
 delete(ER = #ts_entry{}) -> mnesia:dirty_delete_object(ER).
 
-
 list({Username, Timeline}, Start, Length, OrderFun)
 when is_integer(Start) and is_integer(Length) ->
 

src/ts_ext_data.erl

@@ -0,0 +1,47 @@
+-module(ts_ext_data).
+-export([create_table/1, set_property/3, get_property/2, get_properties/1]).
+
+-include("ts_db_records.hrl").
+
+create_table(TableOpts) ->
+    mnesia:create_table(ts_entry,
+        TableOpts ++ [{attributes, record_info(fields, ts_ext_data)},
+                      {type, ordered_set}]).
+
+% set last timeline
+set_property(Ref=#ts_user{}, last_timeline, LastTimelineId) ->
+    do_set_property(Ref, last_timeline, LastTimelineId);
+
+% Set exclusion_list for a User account
+set_property(Ref=#ts_user{}, entry_exclusions, ExclusionList) ->
+    do_set_property(Ref, entry_exclusions, string:join(ExclusionList, "|"));
+
+% Set exclusion_list for a Timeline entry
+set_property(Ref=#ts_timeline{}, entry_exclusions, ExclusionList) ->
+    do_set_property(Ref, entry_exclusions, string:join(ExclusionList, "|"));
+
+set_property(Ref, Key, Value) ->
+    throw(io_lib:format("Property '~s' not available for a ~s record.",
+        [Key, element(1, Ref)]).
+
+get_property(Ref, PropKey) ->
+    {atomic, Result} = mnesia:transaction(fun() ->
+        case mnesia:read(ts_ext_data, {Ref, PropKey}) of
+            [] -> not_set;
+            [Property] -> Property
+        end
+    end),
+    Result.
+
+get_properties(Ref) ->
+    {atomic, Result} = mnesia:transaction(fun() ->
+        MatchHead = #ts_ext_data{ref = {Ref, '_'}, _='_'},
+        mnesia:select(ts_ext_data, [{MatchHead, [], ['$_']}])
+    end),
+    Result.
+
+do_set_property(Ref, PropKey, Val) ->
+    {atomic, Result} = mnesia:transaction(fun() ->
+        mnesia:write(#ts_ext_data{ref = {Ref, PropKey}, value = Val}).
+    end),
+    Result.

src/ts_json.erl

@@ -1,24 +1,25 @@
 -module(ts_json).
--export([encode_record/1, record_to_ejson/1, ejson_to_record/2, ejson_to_record_strict/2]).
+-export([encode_record/1, record_to_ejson/2,
+         ejson_to_record/2, ejson_to_record/3,
+         ejson_to_record_strict/2, ejson_to_record_strict/3]).
 
 -include("ts_db_records.hrl").
 
 encode_record(Record) -> lists:flatten(json:encode(record_to_ejson(Record))).
 
-% User JSON record structure:
+% User JSON record required structure:
 % {"id":        "john_doe",
 %  "name":      "John Doe",
 %  "email":     "john.doe@example.com",
-%  "join_date": "2011-01-01T12:00.000Z",
+%  "join_date": "2011-01-01T12:00.000Z"}
-%  "ext_data":  {"last_timeline", "personal"}}
 
-record_to_ejson(Record=#ts_user{}) ->
+record_to_ejson(Record=#ts_user{},  ExtData) ->
     {struct, [
         {id, Record#ts_user.username},
         {name, Record#ts_user.name},
         {email, Record#ts_user.email},
-        {join_date, encode_datetime(Record#ts_user.join_date)},
+        {join_date, encode_datetime(Record#ts_user.join_date)}] ++
-        {ext_data, Record#ts_user.ext_data}]};
+        ext_data_to_ejson(ExtData)};
 
 % Timeline JSON record stucture:
 % {"user_id":   "john_doe",
@@ -26,7 +27,7 @@ record_to_ejson(Record=#ts_user{}) ->
 %  "created":   "2011-01-01T14:00.000Z",
 %  "description:"Personal time-tracking."}
 
-record_to_ejson(Record=#ts_timeline{}) ->
+record_to_ejson(Record=#ts_timeline{}, ExtData) ->
     % pull out the username and timeline id
     {Username, TimelineId} = Record#ts_timeline.ref,
 
@@ -35,7 +36,8 @@ record_to_ejson(Record=#ts_timeline{}) ->
         {user_id, Username},
         {id, TimelineId},
         {created, encode_datetime(Record#ts_timeline.created)},
-        {description, Record#ts_timeline.desc}]};
+        {description, Record#ts_timeline.desc}] ++
+        ext_data_to_ejson(ExtData)};
 
 % Entry JSON record structure:
 % {"user_id":       "john_doe",
@@ -45,7 +47,7 @@ record_to_ejson(Record=#ts_timeline{}) ->
 %   "mark":         "Workout.",
 %   "notes":        "First workout after a long break."}
 
-record_to_ejson(Record=#ts_entry{}) ->
+record_to_ejson(Record=#ts_entry{}, ExtData) ->
     % pull out the username, timeline id, and entry id
     {Username, TimelineId, EntryId} = Record#ts_entry.ref,
 
@@ -59,68 +61,78 @@ record_to_ejson(Record=#ts_entry{}) ->
         {id, EntryId},
         {timestamp, encode_datetime(DateTime)},
         {mark, Record#ts_entry.mark},
-        {notes, Record#ts_entry.notes}]}.
+        {notes, Record#ts_entry.notes}] ++
+        ext_data_to_ejson(ExtData)}.
 
 encode_datetime({{Year, Month, Day}, {Hour, Minute, Second}}) ->
     lists:flatten(io_lib:format("~4.10.0B-~2.10.0B-~2.10.0BT~2.10.0B:~2.10.0B:~2.10.0B.~3.10.0BZ", 
         [Year, Month, Day, Hour, Minute, Second, 000])).
 
-ejson_to_record(Rec=#ts_timeline{}, EJSON) ->
+ext_data_to_ejson(ExtData) -> ext_data_to_ejson(ExtData, []).
-    {struct, Fields} = EJSON,
 
-    Created = case lists:keyfind(created, 1, Fields) of
+ext_data_to_ejson([], ExtData) -> ExtData;
-        false -> undefined;
-        {created, CreatedVal} -> decode_datetime(CreatedVal)
-    end,
 
-    Desc = case lists:keyfind(description, 1, Fields) of
+ext_data_to_ejson([{Key, Value}|T], Acc) ->
-        false -> undefined;
+    case Key of
-        {description, DescVal} -> DescVal
+        entry_exclusions -> ext_data_to_ejson(T, [{Key, {array, Value}}|Acc]);
-    end,
+        _Other -> ext_data_to_json(T, [{Key, Value}|Acc])
+    end.
 
-    Rec#ts_timeline{
+ejson_to_record(Empty, {struct, EJSONFields}) ->
-        created = Created,
+    construct_record(Empty, EJSONFields, []).
-        desc = Desc };
 
-ejson_to_record(Rec=#ts_entry{}, EJSON) ->
+ejson_to_record(Empty, Ref, EJSON) ->
-    {struct, Fields} = EJSON,
+    Constructed = ejson_to_record(Empty, EJSON),
+    setelement(2, Constructed, Ref).
 
-    Timestamp = case lists:keyfind(timestamp, 1, Fields) of
+ejson_to_record_strict(Empty=#ts_timeline{}, EJSON) ->
-        false -> undefined;
+    Constructed = ejson_to_record(Empty, EJSON),
-        {timestamp, TSVal} -> calendar:datetime_to_gregorian_seconds(
+    case Constructed of
-            decode_datetime(TSVal))
+        #ts_timeline{created = undefined} ->
-    end,
+            throw("Missing timeline 'created' field.");
+        #ts_timeline{desc = undefined} ->
+            throw("Missing timeline 'description' field.");
+        _Other -> Constructed
+    end;
 
-    Mark = case lists:keyfind(mark, 1, Fields) of
+ejson_to_record_strict(Empty=#ts_entry{}, EJSON) ->
-        false -> undefined;
+    Constructed = ejson_to_record(Empty, EJSON),
-        {mark, MarkVal} -> MarkVal
+    case Constructed of
-    end,
+        #ts_entry{timestamp = undefined} ->
+            throw("Missing timelne 'timestamp' field.");
+        #ts_entry{mark = undefined} ->
+            throw("Missing timeline 'mark' field.");
+        #ts_entry{notes = undefined} ->
+            throw("Missing timeline 'notes' field/");
+        _Other -> Constructed
+    end.
 
-    Notes = case lists:keyfind(notes, 1, Fields) of
+ejson_to_record_strict(Empty, Ref, EJSON) ->
-        false -> undefined;
+    Constructed = ejson_to_record_strict(Empty, EJSON),
-        {notes, NotesVal} -> NotesVal
+    setelement(2, Constructed, Ref).
-    end,
 
-    Rec#ts_entry{
+construct_record(Timeline=#ts_timeline{}, [{Key, Val}|Fields], ExtData) ->
-        timestamp = Timestamp,
+    case Key of
-        mark = Mark,
+        created -> construct_record(
-        notes = Notes}.
+                Timeline#ts_timeline{created = decode_datetime(Value)},
+                Fields, ExtData);
+        description -> construct_record(Timeline#ts_timeline{desc = Value},
+                Fields, ExtData);
+        Other -> construct_record(Timeline, Fields, [{Key, Value}|ExtData])
+    end;
 
-ejson_to_record_strict(Rec=#ts_timeline{}, EJSON) ->
+construct_record(Entry=#ts_entry{}, [{Key, Value}|Fields], ExtData) ->
-    {struct, Fields} = EJSON,
+    case Key of
+        timestamp -> construct_record(
+            Entry#ts_entry{timestamp = calendar:datetime_to_gregoraian_seconds(
+                decode_datetime(Value))},
+            Fields, ExtData);
+        mark -> construct_record(Entry#ts_entry{mark=Value}, Fields, ExtData);
+        notes -> construct_record(Entry#ts_entry{notes=Vale}, Fields, ExtData);
+        Other -> construct_record(Entry, Fields, [{Key, Value}|ExtData])
+    end;
 
-    Rec#ts_timeline{
+construct_record(Record, [], ExtData) -> {Record, ExtData}.
-        created = decode_datetime(element(2, lists:keyfind(created, 1, Fields))),
-        desc = element(2, lists:keyfind(description, 1, Fields))};
-
-ejson_to_record_strict(Rec=#ts_entry{}, EJSON) ->
-    {struct, Fields} = EJSON,
-
-    Rec#ts_entry{
-        timestamp = calendar:datetime_to_gregorian_seconds(decode_datetime(
-            element(2, lists:keyfind(timestamp, 1, Fields)))),
-        mark = element(2, lists:keyfind(mark, 1, Fields)),
-        notes = element(2, lists:keyfind(notes, 1, Fields))}.
 
 decode_datetime(DateTimeString) ->
     % TODO: catch badmatch and badarg on whole function
@@ -144,3 +156,15 @@ decode_datetime(DateTimeString) ->
         list_to_integer(SecondString)},
 
     {Date, Time}.
+
+ejson_to_ext_data(ExtData) -> ejson_to_ext_data(ExtData, []).
+
+ejson_to_ext_data([], ExtData) -> ExtData;
+
+ejson_to_ext_data([{Key, Value}|T], Acc) ->
+    case Key of
+        entry_exclusions -> ext_data_to_ejson(T, [{Key, element(2, Value)}|Acc]);
+        _Other -> ext_data_to_json(T, [{Key, Value}|Acc])
+    end.
+
+

src/ts_timeline.erl

@@ -1,5 +1,5 @@
 -module(ts_timeline).
--export([create_table/1, new/1, update/1, write/1, lookup/2, list/3]).
+-export([create_table/1, new/1, new/2, update/1, update/2, write/1, lookup/2, list/3]).
 
 -include("ts_db_records.hrl").
 -include_lib("stdlib/include/qlc.hrl").
@@ -11,8 +11,14 @@ create_table(TableOpts) ->
 
 new(TR = #ts_timeline{}) -> ts_common:new(TR).
 
+new(TR = #ts_timeline{}, ExtData) when is_list(ExtData) ->
+    ts_common:new(TR, ExtData).
+
 update(TR = #ts_timeline{}) -> ts_common:update(TR).
 
+update(TR = #ts_timeline{}, ExtData) when is_list(ExtData) ->
+    ts_common:update(TR, ExtData).
+
 write(TR = #ts_timeline{}) -> mnesia:dirty_write(TR).
     
 lookup(Username, TimelineId) ->
@@ -22,8 +28,6 @@ lookup(Username, TimelineId) ->
     end.
 
 list(Username, Start, Length) -> 
-    {atomic, Timelines} = mnesia:transaction(fun() ->
+    MatchHead = #ts_timeline{ref = {Username, '_'}, _='_'},
-        MatchHead = #ts_timeline{ref = {Username, '_'}, _='_'},
+    mnesia:dirty_select(ts_timeline, [{MatchHead, [], ['$_']}]),
-        mnesia:select(ts_timeline, [{MatchHead, [], ['$_']}])
-    end),
     lists:sublist(Timelines, Start + 1, Length).

src/ts_user.erl

@@ -1,6 +1,6 @@
 -module(ts_user).
-%-export([create_table/1, new/1, update/1, lookup/1, list/2]).
+-export([create_table/1, new/1, new/2, update/1, update/2, lookup/1, list/2,
--compile(export_all).
+         check_credentials/2]).
 
 -include("ts_db_records.hrl").
 -include_lib("stdlib/include/qlc.hrl").
@@ -12,13 +12,35 @@ create_table(TableOpts) ->
 
 % expects the password in clear
 new(UR = #ts_user{}) -> 
-    case mnesia:dirty_read(ts_user, UR#ts_user.username) of
+    {atomic, Result} = mnesia:transaction(fun() -> do_new(UR) end),
+    Result.
+
+new(UR = #ts_user{}, ExtData) ->
+    {atomic, Result} = mnesia:transaction(fun() ->
+        NewUser = do_new(UR),
+        ts_common:do_set_ext_data(UR, ExtData),
+        NewUser end),
+    Result.
+
+do_new(UR = #ts_user{}) -> 
+    case mnesia:read(ts_user, UR#ts_user.username) of
         [ExistingRecord] -> {error, {record_exists, ExistingRecord}};
-        [] -> mnesia:dirty_write(hash_input_record(UR))
+        [] -> mnesia:write(hash_input_record(UR))
     end.
 
 update(UR = #ts_user{}) ->
-    case mnesia:dirty_read(ts_user, UR#ts_user.username) of
+    {atomic, Result} = mnesia:transaction(fun() -> do_update(UR) end),
+    Result.
+
+update(UR = #ts_user{}, ExtData) ->
+    {atomic, Result} = mnesia:transaction(fun() ->
+        UpdatedUser = do_update(UR),
+        ts_common:do_set_ext_data(UR, ExtData) 
+        UpdatedUser end),
+    Result.
+
+do_update(UR = #ts_user{}) ->
+    case mnesia:read(ts_user, UR#ts_user.username) of
         [] -> no_record;
         [Record] ->
             UpdatedRecord = ts_common:update_record(Record, UR),
@@ -26,7 +48,7 @@ update(UR = #ts_user{}) ->
                 undefined -> UpdatedRecord;
                 _Password -> hash_input_record(UpdatedRecord)
             end,
-            mnesia:dirty_write(HashedRecord)
+            mnesia:write(HashedRecord)
     end.
 
 lookup(Username) ->